Kirk's Planet

Kirk's Gartner Blog: The Android Dilemma – Fragmentation

Tue, 03 Apr 2012 14:44:14 +0000

The best thing that Google can do for the Android ecosystem is develop a world-class piece of hardware that runs the Android operating system. Unfortunately, the worst thing that Google can do for the Android ecosystem is develop a world-class piece of hardware that runs the Android operating system. Yeah, that puts Google in a tough spot!

Google’s recent purchase of Motorola Mobility is certainly a patent play, but it also gives Google the luxury of building an Android device should they choose. They’ve already announced they’ll be delivering a Google branded tablet, though they’re partnering with Asus, not using the Xoom. Perhaps this is just the first step. Next up? A Google branded smartphone where they “partner” with Motorola Mobility. All just pure speculation at this point, but still interesting to ponder.

Given the existing fragmentation of the Android ecosystem, this may not be a bad idea. It’s one thing when developers start complaining of fragmentation. It’s yet another when consumers start complaining because apps don’t work. In the end, fragmentation is killing the Android platform and Google has to address this issue ASAP.

With Google developing their own devices, they’ll have complete control over the experience (ala Apple), but it’s sure to drive away device manufacturers who have built atop Android. Hey, maybe webOS isn’t dead. After all, it’s a great mobile operating system that’s now open source. More on that another day.

If Google chooses not to build their own smartphone, they have to take at least two important steps. First, they have to gain control over updates to the Android operating system. Leaving updates in the hands of network operators and device manufacturers is the cause for this fragmentation. Second, they have to ensure that device hardware profiles created today possess the ability to run future versions of Android for at least two years (ie. the length of a typical network contract). Preferably longer.

If Google doesn’t take one of these two steps (i.e., their own device OR gaining more control) fragmentation on the Android platform will persist. And once users start suffering, that’s not a good thing for Android. In the meantime, if you’re shopping for an Android device, buyer beware. You can still purchase a device running Android 2.1 (aka. Eclair), which first hit the market in 2010 and is over two years old. If you happen to purchase one of these devices, don’t expect an upgrade to Ice Cream Sandwich (v. 4.0). And if you’re developing for the Android platform..well…good luck with that too. You’d best understand market share and try limiting your options.

@kirkk.com blog: Java With a Bit of OSGi - The Book

Mon, 26 Mar 2012 14:09:00 +0000

I’m dancing. By god I’m dancing on the walls. I’m dancing on the ceiling. I’m ecstatic. I’m overjoyed. I’m really, really pleased.
- An excerpt from the Foreword by Uncle Bob (aka. Robert C. Martin)

My book, Java Application Architecture: Modularity Patterns With Examples Using OSGi is now available. Uncle Bob and Peter Kriens each contributed Forewords to the book. The book itself is part of the Robert C. Martin series. The book is intended for all software developers interested in designing better software using modularity. Though the examples use Java, the techniques can be applied to other languages and platforms, such as .NET, with relative ease.

Even if you’re not using OSGi (or perhaps not even familiar with OSGi), I’m confident you’ll find the book valuable. The book (and patterns) has been designed to allow you to realize the benefits of modularity whether you’re using a module framework, or not. As Uncle Bob says in the Foreword, “This is how you build a Java application, people.” Peter sums it up nicely too, in saying “This book…will give you a view into the magic of modularity.”

You can order it online at Amazon (print edition & Kindle edition), iBooks, InformIT, or a number of other publishers. For more details on the book, please see the book’s website. Over the next couple of weeks, I plan to post a sample chapter or two that will give you a feel for the book’s contents.

Here’s what a few people have to say:

“The fundamentals never go out of style, and in this book Kirk returns us to the fundamentals of architecting economically interesting software-intensive systems of quality. You’ll find this work to be well-written, timely, and full of pragmatic ideas.” – Grady Booch, IBM Fellow
“Along with GOF ‘Design Patterns’ - ’Java Application Architecture’ is a must own for every enterprise developer and architect, and on the required reading list for all Paremus engineers.” – Richard Nicholson, Paremus CEO & President of the OSGi Alliance
“In writing this book, Kirk has done the software community a great service: he’s captured much of the received wisdom about modularity in a form which can be understood by newcomers, taught in Computer Science courses, and referred to by experienced programmers. I hope this book finds the wide audience it deserves.” – Glyn Normington, Eclipse Virgo Project Lead
“Our industry needs to start thinking in terms of modules – it needs this book!” – Chris Chedgey, Founder and CEO of Structure 101
“In this book Kirk Knoernschild provides us with the design patterns we need to make modular software development work in the real world. While it’s true that modularity can help us manage complexity and create more maintainable software, there’s no free lunch. If you want to achieve the benefits modularity has to offer, buy this book.” – Patrick Paulin, Consultant and Trainer at Modular Mind
“Kirk has expertly documented the best practices for using OSGi and Eclipse runtime technology. A book any senior Java developer needs to read to better understand how to create great software.’ –Mike Milinkovich, Executive Director Eclipse Foundation

I’d like to thank all of you who helped me along this journey. I hope you enjoy the book.

Kirk's Gartner Blog: A Mobile Kata

Mon, 26 Mar 2012 13:14:00 +0000

A programming kata is a programming exercise that you repeat many times over. A programming kata is useful for a few different reasons.

It helps you learn a new programming language or framework.
It helps you hone your skills with a language, framework, or tool you already use.
It can provide fresh insight on how you might use your existing tools by applying techniques you’ve learned using an alternative tool.
It can help you evaluate and compare different frameworks and tools.
It can serve as a reference implementation you use to teach others.

A kata isn’t exclusive to just programming, however. In fact, the term kata originated in karate, and you can use the concept of a kata to help you learn new tools, refine processes, or just about anything else you want to get better at.

One kata I’ve applied numerous times is my Loan kata. The Loan kata accepts an amount, term, and rate and calculates the monthly payment for a loan. Variations of the kata calculate the amount each month applied to prinicipal and interest. Over the next several weeks, I’m going to use the Loan kata to develop different types of mobile applications. Each time, I’ll post a blog entry that illustrates what I’ve done and shares some of my findings. Right now, I intend to do the following:

Create a mobile web app using JQuery Mobile.
Create a mobile web app using Sencha Touch.
Create a resident mobile app using PhoneGap.
Create a resident mobile app using the iOS SDK
Create a resident mobile app using the Android SDK
Create a resident mobile app using BlackBerry SDK
Create a resident mobile app using Windows Phone SDK

If you’re interested in seeing another tool or framework used, let me know and I’ll add it to the list. I don’t have a timeline to complete all this, and it will take me several weeks to pull them all together. But stay tuned for some programming fun!

Kirk's Gartner Blog: Traceability – Give Me a Break!

Tue, 13 Mar 2012 15:16:00 +0000

Today, I got an e-mail about a webinar on requirements traceability. Come on, give me a break! Conceptually, the goal of traceability is a noble one. The ability to link artifacts back to stakeholder needs and forward to the design artifacts, source code and tests that realize those requirements help us, among other things, understand and assess the impact of change. But there’s some hidden challenges that blow traceability right out of the water.

There are two key requirements with traceability.

Related artifacts must be linked together.
Linked artifacts must maintain fidelity.

Every software artifact is an expression of some requirement. On a typical software project, a single requirement is expressed several different ways (use case, business rules, user story, test case, class diagram, source code). But regardless of how it’s expressed, it must convey the same meaning across all linked artifacts. That is, it must possess fidelity.

Establishing the links between artifacts is the easy part. Managing these relationships going forward is virtually impossible. We can only maintain fidelity across artifacts if we synchronize the artifacts anytime something changes. We all know that this doesn’t happen. When a test plan changes, we rarely go back and update requirements documentation. When the source code changes, we rarely update the design document.

The primordial force that makes traceability unattainable is the reliance on the power of human discipline to ensure that the correct links between artifacts are established and that each artifact maintains fidelity with other linked artifacts. The more things change, the more discipline that’s required. Most teams simply don’t have the time to maintain that discipline. The result is the outdated and irrelevant documentation that plagues many projects today. Again, establishing the links between artifacts is easy. Ensuring linked artifacts are synchronized and maintain fidelity is hard.

Many tools claim to offer support for full lifecycle traceability. Be careful. What many offer is the possibility to establish links between artifacts, with no guarantee that the correct links have been established and that the artifacts offer a consistent expression of a requirement. Traceability without fidelity will not make you happy.

So, is traceability attainable? Absolutely. But only if related artifacts are linked and the artifacts convey the same meaning. Since we don’t possess the discipline to do this manually, the only way to ensure both is through executable artifacts. Executable artifacts are artifacts that are executable against another artifact. Executable artifacts establish a link and enforce fidelity. The simplest example is an automated test against the code. If the code changes, the test may break, resulting in a lack of fidelity. Fortunately, this lack of fidelity raises an error condition and make it visible to the team, forcing them to fix the problem (i.e., synchronize).

Until tools support executable artifacts, the promise of traceability will go unfulfilled.

If you’re a Gartner for Technical Professionals client, you can read more about this idea in our SDLC Reference Architecture and Requirement Management template.

Kirk's Gartner Blog: Mobility, Architecture, & Code

Thu, 01 Mar 2012 20:02:00 +0000

It’s been a while, but it’s good to be back. After a year-long hiatus, I’m back in the blogging saddle. This first post is just a quick “hello” and a shout out to let everyone know what you’ll find on my little corner of the Gartner Blog Network. You’ll find me discussing three things specifically:

Mobile Development: Most of my research these days is focused on mobile application development. This includes the mobile web, cross platform frameworks, and the native software development kits (SDKs). If, like me, mobile development is your bag, you might find the the following Gartner documents a good place to start – Mobile Web Applications, Cross Platform Mobile Development Frameworks, and Mobile Applications: Native, Cross-Compiled, Custom Container, Hybrid, or Web.

Software Architecture: For several years, I’ve taken a keen interest in software architecture. In fact, this interest is one of the reasons I haven’t been blogging as much lately. I’ve been working on a separate writing project that is just wrapping up. I’ll make an announcement on this soon. But for those of you already familiar with some of the content on my previous blog, rest assured you’ll find similar content here. I mean seriously…it’s been way too long since I’ve written anything about OSGi, don’t you think?

Code: Through all the principles, practices, patterns, disciplines, and methodologies we discuss ever so frequently, none of it would matter without the code. Seriously, it’s pretty tough to have a software system without the code. Since I still stay pretty close to the code, taking a bit of time each week to hone my skills, you’ll find my posts often include a bit of code to drive a point home or simply demonstrate how to do something.

Oh, I suppose every once in a while I’ll stray from these three topics and discuss the software development process, a programming language or two, or just technology in general. It’s a geeks world, you know. If these topics interest you, why not subscribe to my feed and participate in the discussion.

I’m excited to be back blogging again.

Kirk's Gartner Blog: The Prime Directive of Software Development

Thu, 12 Aug 2010 14:00:15 +0000

“Forward and sustainable progress can only be measured through a software system that works. There is no other way!”

Those are the words I uttered at our recent Catalyst conference in San Diego when debating the merits of agile development on stage with a colleague.

Requirements documentation, architecture and design models, and test plans may provide value, but they do not offer accurate insight into the current state of the project. While we may value these artifacts, we must value working software more. Without a system that works, there is no way to be certain that the development team is moving in the right direction. Continuing to develop and add new features to a software system that doesn’t compile, or whose tests fail, will only steer the development team further off course, away from their intended destination. Without question, it is counterproductive to add code to a system that doesn’t work. A software system with a multitude of compilation errors or failing tests only offers a false sense of security that the team is making progress.

The prime directive of software development states that a software system should always compile and its tests should always execute successfully.

From the moment the first line of code is written to the moment the team gathers for the “go/no go” meeting, and everywhere in between…the system must work. The software development team should strive to ensure the system can be built, tested, and executed at any point in time throughout the development lifecycle.

The prime directive assures the development team that they are building a system exhibiting the functionality their users demand. It ensures they are developing a system that performs, is secure, and is dependable. How so? The system is always functional, so it can be tested throughout the lifecycle. Customer demonstrations can be held that ensure the software meets the needs of the users. Important feedback garnered from these demonstrations allow the development team to adjust their course. Problems aren’t allowed to fester in the system for prolonged periods of time. As issues arise, they are dealt with by the development team. The result? No problem is ever older than the last successful build.

Of course, the prime directive has implications. If at any point in time the software is broken, it must be the priority of the development team to resolve the situation as quickly as possible. The software should never be left broken for an extended period of time. This can seem counterintuitive, especially for teams with severe deadline pressures. The prime directive demands discipline. And it is with this discipline that the development team is able to ensure sustainable and forward progress throughout the duration of a project.

There are many practices, agile and otherwise, that development teams adopt to improve quality, speed delivery, and increase project transparency. But each of these practices depends upon the fundamental notion that the software works. That is how it should be. That is how it must be!

Kirk's Gartner Blog: Continuous Integration

Wed, 04 Aug 2010 18:04:38 +0000

Agile transitions are tough. Real tough! But there are some important practices a team can employ to increase agility without undergoing a massive agile transition effort. One of these practices is Continuous Integration (CI), and I’m hard pressed to imagine a software development team that wouldn’t benefit from a sound CI strategy. After all, if we cannot prove that a software system works, we must question if we’re truly making progress by adding code to a broken system. In fact, leveraging CI is one way to launch an initiative to increase the agility of a software development team.

On the Test Early blog, there’s a great little parody that offers a play on the proverbial rhyme, “For Want of a Nail”, which illustrates how even the smallest of actions (or lack thereof) can have significant consequences.

FOR WANT OF A BUILD

For want of a build, a test case was not executed
For want of test case, a defect was not detected
For want of a defect report, a bad release was promoted
For want of a good release, a strategic customer was lost
For want of a customer, a development team was reduced
For want of developers, a product stagnated
For want of a product, a company was lost
And all for the want of a build…

Source: http://www.testearly.com/2007/08/07/for-want-of-a-nail/

While the result as played out here is pretty extreme, the essence of CI is captured quite well. Emphasizing quality throughout the development effort, in lieu of attempting to verify quality at the end, will likely result in a much higher quality product. In other words, Build Quality In. CI helps us accomplish this feat through frequent builds and test execution that allows us to verify the system is always in a working state. Additionally, other important lifecycle activities are enabled that provide the important feedback necessary to ensure we never stray to far off the intended course.

Kirk's Gartner Blog: Software Development & the Technology We Use

Fri, 11 Jun 2010 16:14:44 +0000

It’s exciting to be part of GBN, and I look forward to sharing opinions with each other. Make no mistake…this is a technology blog. But I hope to blog about technology topics in a way that helps folks understand the strengths and weaknesses of the technology, as well as helping them make the important decisions surrounding the technologies they’ll use going forward. I hope you’ll join me in exploration, and contribute your views.

Here’s a bit of what you can look forward to reading about in my little corner of the GBN world, along with my brief views on each. Over time, we’ll delve into each more deeply.

Software Development Processes and Practices

I’m an ardent believer that agile development methods and practices are *always* good, they do scale, and are a critical success factor for large teams. While that may sound overly zealous, I cannot think of any situation where slow, brittle, and resistance to change are beneficial attributes. Granted, my definition of agile may be slightly different from others, so this will be fun to explore.

Platforms, Development Frameworks, Software Architecture, and Modularity

The application platform that we’ve grown accustomed to over the past decade is undergoing transformation that stands to affect everyone from the developer to the folks in the data center. There is a lot of buzz surrounding the cloud, but there is also significant innovation elsewhere that promises to dramatically alter the tools we use to develop software. Dynamic languages that promise improved productivity and functional languages that promise improved performance are two examples. First class support for modularity (e.g., OSGi) that brings with it platform componentization and the opportunity to rightsize the platform is another. And each has a significant impact on software architecture of the future.

Rich Mobile Application Platforms

By now, I suspect you are familiar with these new breed of mobile devices garnering so much attention. Hardly a day goes by without an earth shattering announcement. The battle between Apple and Google is exciting to watch. Yet beneath these very public disputes lie an interesting set of dynamics. For instance, is Apple’s curated ecosystem really the devil? Or might the fragmentation of the Android platform be a bit more concerning? And each affects how you develop rich mobile applications for the respective platforms, albeit in different ways.

Presentation Technologies

In the mid-1990′s, rich client technology was the rage. Remember PowerBuilder, anyone? Of course, the web changed all that, offering a near ubiquitous platform for delivering the corporate brand to consumers. Unfortunately, web technologies presented new challenges. Rich internet application technologies (RIA) such as Ajax and Flash helped overcome some of the user experience problems, but others persisted. The lack of local storage and inability to access applications in the absence of an internet connection are two examples. While new technologies such as Silverlight and AIR promised to finally overcome these challenges, HTML 5 aims to do the same. And on the horizon looms a rather interesting battle that will redefine the web.

While the above topics are my primary interest, I cannot promise that I won’t occasionally stray and discuss other topics, as well. It should be fun.

@kirkk.com blog: Farewell Blog, Hello POMA

Fri, 07 May 2010 17:18:53 +0000

As the saying goes…all good things must end. As of today, my blog is shutting down!

I do intend to leave all content on-line so you’ll always be able to see a list of all 134 posts, but there won’t be any new posts for the foreseeable future. I’d like to thank everyone that has taken the time to read my often times long-winded posts. I hope you’ve found them useful.

For the past few years, I’ve written almost exclusively about modularity and OSGi (with a few other topics sprinkled in on occasion). To that end, I’ve decided to pursue a long overdue book project, tentatively titled Patterns of Modular Architecture (POMA). I hope you’ll decide to join me in my journey. Also, be sure to follow the book updates on Twitter.

So as to not disappoint those who have come to expect something a bit more verbose, I have managed to cobble together a few words explaining my decision in more detail. So if you want to know more, read on!

Story Behind the Story

Over the past few years of blogging, I’ve focused considerable energy on espousing the virtues of modularity and OSGi. From conceptual posts that discuss the important architectural benefits of modularity to simple examples that illustrate the benefit. From the beginning to the end, I’ve written about the benefits of modular architecture at a conceptual level, practical level, and have even provided some concrete examples. To the chagrin of some, I’ve even discussed the challenges that lie ahead. Of course, modularity and OSGi aren’t the only topics I write about. I’ve also written about agility, IT labor, metrics, and more. At some point though, the topics always came back to modularity and OSGi.

In the time that I’ve been focused on modularity and OSGi, the number of folks that access the content on this site have gone from less than 1000 visitors per month to more than 10,000, culminating in more than 100,000 pages served up and over a quarter million hits to the site each month. The blog was listed as one of the Top 200 Blogs for Developers, a Top Analyst Blog, and many of my posts are syndicated on JavaLobby. While still small by many standards, I recognized progress. And in general, the feedback I’ve received from the community is positive, though there is no doubt that at times, I’ve struck some nerves.

For those that read my posts, you already know that most tend to span many paragraphs. They require extensive writing time, considerable editing time, and careful review. It’s taken me hours to author many of them. And all of the content on this site has always been licensed under a Creative Commons Attribution-Noncommercial-Share Alike license. I vowed to never include any noise on the site, such as Google AdSense. It’s always been an earnest attempt to help developers improve the systems they create. A labor of love, you might say.

A while back, I introduced the modularity patterns, but didn’t elaborate much on the patterns themselves. Alas, that time has come. I’ve quickly realized that I do not have the bandwidth to continue with my long-winded posts and write a book. I’ve decided to pursue the latter and am hopeful that a book on modular architecture will have a far greater, longer lasting impact that serves as a capable ambassador for modular architecture. So while my blog may be silent, I have every intent to continue my advocacy of modular architecture, albeit in a way that I hope is more impactful.

I’ve decided to adopt an open and transparent approach to writing the book. The book’s website illustrates the book in it’s current form - a rough and incomplete draft to be sure. But I hope you’ll take the time to check it out, and offer any feedback you might have. I’ve created a Reviewers page that provides some guidance on the type of feedback I hope to receive. Also, be sure to follow the updates on Twitter.

By the way, this does not mean that I’ll necessarily cease my blogging activities altogether. In the past, I’ve published some articles to the APS Blog. Soon, the APS blog will also be shut down, and we’ll begin to blog on the Gartner Blog Network (GBN). So be sure to checkout GBN on occasion!

Thank you!

@kirkk.com blog: Ecosystems, Modularity, & OSGi

Mon, 03 May 2010 15:50:14 +0000

Recently, I questioned whether OSGi and modularity would succeed in penetrating the enterprise. But what I really meant to question is whether OSGi will have the disruptive impact of which it’s capable. I asserted that if OSGi does succeed, it won’t be based on the technical merits of OSGi. It’ll be because of something else. Something trendy. Something fashionable.

Now, I could be wrong. OSGi adoption is certainly increasing, albeit slowly. And as case studies begin to emerge that tout the cost reduction, improved responsiveness, and time-to-market advantages of OSGi, adoption will likely continue to rise.

But adoption is one thing, disruption another, and I still have this nagging sensation that serves as cause for pause. What if something trendier, more fashionable surfaces, and OSGi is pushed into the backwaters? Will it really have the impact it’s capable of? You know…an “iPhone-esque” impact that raises the bar and redefines an industry. Not just evolutionary, but truly disruptive.

OSGi As Enabler

For OSGi to cross the chasm, it must enable something big that a business wants to buy. Maybe cost reduction, improved responsiveness and time-to-market benefits will be enough. But that’s easily perceived by many as a rather evolutionary impact. Not really disruptive.

Quite possibly OSGi will flourish in the data center, as organizations seek more adaptable platforms that lend them these benefits. But this doesn’t necessarily guarantee that development teams will leverage OSGi to build systems with a modular architecture. Because leveraging a platform built atop OSGi is separate from building modular software systems, even though OSGi enables both.

Without something trendy that promises real benefits, it’ll get brushed under the carpet like what has happened to many technologically superior solutions. Perhaps it’s already happening, with all of the hype surrounding the cloud. Or maybe it’ll be the cloud that helps OSGi cross the chasm. OSGi doesn’t obviously enable the cloud, but OSGi can enable the dynamic and adaptive runtime benefits of the cloud.

But again, this doesn’t mean that teams will begin leveraging OSGi to design modular software. It only means that the platform itself is adaptable. Of course, there is benefit in that. But if that’s the route that is taken, teams will still continue to develop monolithic applications that lack architecturally resiliency, and the full benefit of OSGi (and modularity) will not be realized.

The Disruption

OSGi has the potential to have have a much broader impact, affecting everyone from the developer to those working in the data center. So what might this trend be that will propel OSGi to stardom?

OSGi enables ecosystems!

Now, you’re thinking I’ve gone on the deep end, perhaps? But give me a chance…let me explain.

A Bit of [Recent] Platform History

To start, I want to take a brief walk down memory lane. Not too far back though, but far enough so that we can see how important the ecosystem is in today’s most successful platforms. And these platforms span a range of markets, from mobile to social media. But each are successful in large part due to a thriving ecosystem.

Apple

In 2007, Apple released their first generation iPhone. Without question, the device revolutionized the mobile phone industry. While the device offers a great user experience that has certainly played a role in its surging popularity, people flock to the iPhone today because of the wealth of applications available. Yeah, “there’s an app for that.”

Apple, recognizing the power of this ecosystem, now delivers the iPad. With fewer preinstalled applications than the iPhone, Apple is counting on the ecosystem to drive adoption. The more consumers who flock to the device, the more developers who flock to the platform to deliver applications. As more applications become available, consumers will buy more iPads. The ecosystem fuels itself. Apple has simply provided the environment for the ecosystem to thrive.

Eclipse

In 2003, the Eclipse team was thinking of ways to make Eclipse more dynamic. Their decision to use OSGi to create a rich client platform that supports plug-in architecture was the first step toward the resulting ecosystem we know today. One of the reasons developers use Eclipse is because there are an abundance of plug-ins available that allow them to do their jobs more effectively. Other developers create Eclipse plug-ins because Eclipse is a popular IDE used by many developers. Again, the ecosystem fuels itself. The Eclipse team provided the environment that allows today’s Eclipse ecosystem to thrive.

If you’re interested, you can read more about the history of Eclipse and OSGi.

Hudson

Arguably, Hudson is today’s most popular continuous integration server. But it hasn’t always been. Before Hudson was CruiseControl. And while CruiseControl did help development teams get started on their path toward continuous integration, it was also unwieldy to use in many ways. With Hudson’s plug-in architecture, developers have the ability to extend the tool in ways the original creator couldn’t imagine or couldn’t find the time to do himself. Kohsuke created Hudson and gave the development community a new platform for continuous integration. With its plug-in architecture though, he also provided an environment that allows the Hudson ecosystem to thrive.

Social Media

Facebook. MySpace. Twitter. Each are examples of social media tools with a strong developer community that creates extensions to the platform that users can leverage to enhance the experience. Facebook Developers. MySpace Developer Platform. Twitter API. Each allows the ecosystem to thrive.

And Others

These are just a few examples. It’s easy to find other platforms with similar ecosystems, as well. The ease with which Wordpress themes and plug-ins can be developed and used to enhance a Wordpress website is another example. In fact, many content management systems provide similar capabilities. A large reason why the Firefox web browser has emerged as the preferred web browser is the ease with which add-ons can be installed that extend the capabilities of the browser. The Atlassian Plugin Framework is another example that uses OSGi, and platforms such as Force.com and SharePoint have built (or are trying to build) similar ecosystems.

The Power of Ecosystems

Aside from Eclipse (and Atlassian), none of these other platforms leverage OSGi. Yet each are wildly successful because of two reasons:

An environment was created that allowed an ecosystem to form and flourish. This environment includes a platform and a marketplace.
A group of customers and developers converged on the marketplace and fueled growth of the platform. The result is a self-sustaining ecosystem.

If you look at many of the more popular platforms that have emerged over the past decade, they tend to possess a similar characteristic - a community of individuals dedicated to providing great solutions leveraging the foundation of the platform. OSGi and modularity enables ecosystems on the Java platform.

Ecosystems and the Two Faces

I’ve already talked about the two faces of OSGi - the runtime model and the development model. I’ve also explained how one could possibly see widespread adoption while the other has little impact. A strong ecosystem surrounding OSGi and modularity must leverage both. Developers would create reusable modules, implying they are designing modular software. For development teams to leverage these modules, they must be using a platform that supports the runtime model.

CBD Has Already Had Its Day, You Say?

Now some of you might be arguing that this sounds a lot like the component based development (CBD) fad of the 1990’s. True…to an extent. Certainly these ideas are not new. But there are also some striking differences between that which OSGi enables and the CBD fad that has come and largely gone, or whose promise was never fully realized.

Foremost, the CBD fad was focused almost exclusively on visual components, such as ActiveX. While some attempted to create components for the Java platform, the movement largely failed to go mainstream. Instead, Java EE grew in popularity and for a number of years, garnered everyone’s attention. Why did this happen?

IMHO, the answer is fairly simple. Even though numerous marketplaces emerged that allowed the consumer and producer to come together to buy and sell components, there was never a suitable component execution environment. That is, an environment that would support dynamic deployment, support for multiple versions, dependency management, and, in general, complete control over all components currently executing within the environment. ActiveX components did have an execution environment (though did not support each of these capabilities), but Java did not. Today, in OSGi, Java has the requisite execution environment!

The Ecosystem

It’s easy find holes in this idea. To explain why it cannot work. Yet, it’s happening elsewhere, so why not on the server…in the enterprise? Certainly there are a variety of different ways such an ecosystem could manifest itself. Possibly multiple ecosystems emerge like what we see in the mobile market today.

But for a moment, imagine the world where you have the ability to easily assemble a platform from pre-built infrastructure modules that exactly meet the demands of your application. You might purchase these modules, you might choose to use open source modules, or you might build them yourself. For those you don’t build, you obtain from a module marketplace, possibly deploying them to your [cloud] environment.

And when you choose to use a module, it’s dynamically deployed to your environment. The modules it depends upon? You’re given the option to purchase and deploy them, as well. You develop your software modules using the sound principles and patterns of modular design to ensure loose coupling and high cohesion. As you roll out your business solution modules, you simultaneously deploy the additional infrastructure modules that are needed.

In this marketplace, modules are sourced by multiple vendors. Some large. Some small. Neither the stack, nor your applications, are monolithic beasts. Instead, they are a composition of collaborating software modules. Your infrastructure isn’t necessarily tied to a specific vendor solution. The option always exists for organizations to purchase modules from different providers, easily swapping one provider module out with another.

The ecosystem flourishes. Developers flock to sell their latest creation. Organizations seek to add amazing capabilities to their rightsized environment at a fraction of the cost compared to what they are accustomed to today. The business benefits are real. The technical advantages are real. And the resulting ecosystem is sustainable.

A successful ecosystem demands both the runtime model and development model. And today, OSGi is the only standard technology that will allow this type of successful ecosystem to form on the Java platform. But will it happen? We may have a ways to go, but it sure would be cool! And it would be a shame if we lost this opportunity.

Thoughts?

Note: If you’re interested in ecosystems more generally, you might want to see the great TED talk by Dan Barber, “How I fell in love with a fish.” It’s very entertaining, informative, and worth 20 minutes of your time. His discussion on sustainability is truly fascinating!

Image Source: http://en.wikipedia.org/wiki/File:Blue_Linckia_Starfish.JPG

@kirkk.com blog: OSGi Perspectives

Wed, 28 Apr 2010 16:52:54 +0000

It seems my recent post on OSGi has ruffled a few feathers. I’ve also received a few personal e-mails suggesting that I have missed the mark because OSGi is a significant component of the next generation application platform. Major vendors such as IBM are building OSGi into their products, and other leading edge vendors leverage OSGi as an enabling technology upon which their technical architecture is built.

My intent wasn’t to indict OSGi, so I was pretty surprised with the defensive responses I received…it’s a great technology and the benefits of modularity are real. As I’ve mentioned, and as many already know, I’m an ardent supporter of OSGi and modularity.

Based on the type of the feedback I’ve gotten on my previous post though, I’ve come to the realization that various constituencies have different perspectives that is contributing to the perception of OSGi adoption. Let me explain.

Differing Perspectives

In my post, The Two Faces of Modularity & OSGi, I talk about the runtime model and the development model. There are significant advantages to each. But interestingly, it’s possible for one to succeed while the other never sees widespread adoption. That is, OSGi might lie at the core of the next generation platform, but it doesn’t necessarily imply that enterprise developers will leverage it to build applications with a modular architecture.

Today, most vendors emphasize the runtime model, touting the cost savings, improved responsiveness, and flexibility of a dynamic platform. So I can understand why a lot of folks have reached out to explain to me the error of my ways. From their perspective, OSGi is flourishing.

An excellent example is the Eclipse platform, which released Eclipse 3.0 in June of 2004. However, I’d wager that only a fraction of Java developers using the Eclipse IDE know that OSGi is the major enabling technology upon which Eclipse is built. Instead, they use Eclipse, install plug-ins, but know very little about the underpinnings of the platform.

From a development perspective I’m incredibly interested in the benefits of modular architecture, and I want tools that reduce the accidental complexity of the development model. However, these tools don’t exist today, and developing enterprise server-side Java software applications leveraging OSGi is painful. So while vendors are leveraging OSGi for various platform benefits, as long as they encourage the traditional deployment model and the dearth of tools persist, enterprise developers will not use OSGi to build applications with a modular architecture. We don’t have it today, and we won’t see it in the future until this changes.

Parting Thoughts

I do believe that OSGI has the potential to be a major component of the next generation application platform. Though there are no guarantees, and I still have concerns. Let’s take SpringSource as an example. After seeing almost zero adoption of dm Server, they donated it to the Eclipse Foundation. And the recent VMForce announcement makes nary a mention of OSGi. Instead, tc Server provides the execution environment. So while some might argue that the runtime platform benefits of OSGi are real, there is a far more significant trend called the cloud that provides many of the same advantages. And the cloud certainly doesn’t need OSGi. Though OSGi could thrive in the cloud.

I’m also not convinced that the benefits of modularity will be realized by enterprise development teams. Will enterprise developers leverage OSGi and modularity as a core component of next generation architecture? I hope so, but I don’t believe it’s the slam dunk that many believe it is. Modularity is important. The benefits are real. It helps teams overcome a serious challenge when developing large enterprise software systems. But we need more developer advocacy surrounding OSGi and the benefits of modular architecture. Software is a fashion industry, and we must attach modularity to something trendy and more aggressively advocate adoption.

It is what it is. And unfortunately, that’s just the way it is.

@kirkk.com blog: The Exciting Enterprise

Mon, 26 Apr 2010 15:25:03 +0000

Developing enterprise software is hard work. Bureaucracy. Large teams. Legacy systems. The list of challenges is immense. But enterprise development can also offer tremendous opportunity and reward. It can even be fun! I hope that at some point, you have the opportunity to experience this feeling.

The Exciting Enterprise

I worked here once. In the exciting enterprise. Working here was different. It was actually exciting. What made it different? Well, it was the process. Exciting process? What’s that? It’s the process that makes software development fun again, productive, meaningful, and satisfying.

So what kind of process is this, you ask? Iterative? Agile? RUP? XP? Scrum? Kanban? Hell no! We never talked process lingo. We didn’t care how agile people thought we were. We didn’t want to get caught up in the bureaucracy and political mayhem surrounding software process improvement. There was work to be done. That’s what we talked about. And then we did what we needed to do to get it done.

Not Always Easy

Working in the exciting enterprise wasn’t always easy though. It required stamina, determination, and discipline. There were lots of people who didn’t appreciate how we went about our business. We didn’t submit weekly status reports. We didn’t have the Gantt chart. Actually, we didn’t have a lot of things. But we had what we needed. To us, it all seemed like common sense. Of course, we had a very experienced team, and we knew what worked and what didn’t.

Plan driven? Predictive? Estimates? Models? Oh yeah…we had it all. We had a 18 month project road map. It was quite a plan. Took a small group of us about 1.5 days to develop. All on a single spreadsheet. Showed all the major systems that we’d retire right along with the new functionality that was going to come online. Probably could have gotten another three months of project time if we wanted to do more planning. But we had code to write.

Oh That Build

Code? But what about the requirements? Yep. Gathered them as we went. And when developers had questions, they asked them. Right to the customers face too. Blasphemy! The business analyst working with the developers. Those BAs were awesome too. They worked real hard to clear any confusion. Made sure developers always had what they needed. And we got ‘em as quickly as we could handle them. A steady stream of requirements flowing in and right back out as an executable piece of software.

Oh sure, it wasn’t always easy. We had lots of important checkpoints along the way to make sure we were on the right track. Weekly checkpoints. Daily checkpoints. Hourly checkpoints. Developers and customers sitting in a room together to see what we’d gotten done the past week. Continuous deployment to an environment where folks could actually use the software. Hourly builds that made sure we never strayed too far from a working system.

I still think about that build. Actually, it was more than a build. It was a piece of frickin’ art. That build was the coolest piece of process I’ve ever seen. It was staged. It was fast. It did pretty much anything we asked it to do. It never got in our way. It just went on it’s merry way, building our system. Hourly. Every hour. Automatically. It was the glue that held the team together as we grew in size from a fledgling crew of six developers to upwards of 100 at times.

We protected that build. When somebody cause it to fail, they’d feel the wrath. Eventually, they all grew to love what it could do. Honestly, what choice did we have if we wanted to ensure we could have weekly checkpoints throughout the process? The only way to pull it off was to emphasize software that works. It always had to work. ALWAYS! Guard the source. The prime directive - software that works!

The Source

Now don’t get me wrong. We had documentation. We had lots of documentation. Some of it was pretty nice looking stuff too. But we weren’t afraid to let it go. Let it serve it’s purpose and then move onto what was real important…the source code. What else really matters, anyway? All the pretty documents, models and plans don’t amount to a hill of beans if the source code isn’t provably correct. That was our focus. It was the teams focus. And we guarded that source code very closely.

Improving the Right Process

I look back now. An 18 month project. A team that clicked. Not just a development team. A team. Including the customers. Honestly, we didn’t just build a piece of software, though. We improved the process. We automated it. We optimized it. No, I’m not talking about the software process. I mean the business process! That’s what made it so cool…

@kirkk.com blog: Granularity: Architecture’s Nemesis

Thu, 22 Apr 2010 18:05:36 +0000

Granularity is the extent to which a system is broken down into it’s behavioral entities. Coarse-grained entities tend to be richer in behavior than fine-grained entities. Because coarse-grained entities do more, they tend to be larger than fine-grained entities, and are easier to use. In general, a fine-grained entity is more reusable than a coarse-grained entity because it does a little bit less. This is pretty natural. If it does less, then it’s more likely to apply across a wider variety of usage scenarios. I spent some time discussing this tension in Reuse: Is the Dream Dead?

Unfortunately, while there are a lot of patterns, principles, and idioms that offer really good guidance surrounding many aspects of architecture and design, there isn’t a lot of guidance surrounding granularity. Let’s start with a really simple example that illustrates the challenge.

A Really Simple Example

Consider the following save method:

public class Person {
   private String firstName;
   private String lastName;
   private String ssNumber;
   public void save() {
      ValidationErrors errors = this.validate();
      if (errors != null) {
         //handle the errors somehow
      } else {
         //save to the database.
      }
   }
   private ValidationErrors validate() {
      //perform validation
   }
}

As can be seen, the save method invokes the validate method before saving the information to the database. Generally speaking, this makes saving the information easier (and possibly safer) since whatever invokes save doesn’t have to worry about validating. And this seems to be a good thing.

Yet, what happens when a new consumer of this method wants to save the information but apply a slightly different set of rules than the current validation method provides? Here’s where the existing save method is simply too coarse-grained. It does just a little bit too much. We’ve made it easier to use, but also less reusable.

There are quite a few variations I could go with at this point to improve the design. One would be to make the validate method public, and have clients invoke the validate method before invoking save. Then, if validate didn’t do what it was supposed to, clients could always opt out of invoking the validate method and apply their own validation before invoking save. Certainly less safe though because validation is optional.

Yet another alternative might be the following:

public class Person {
   private String firstName;
   private String lastName;
   private String ssNumber;
   public void save(Validator validator) {
      ValidationErrors errors = validator.validate();
      if (errors != null) {
         //handle the errors somehow
      } else {
         //save to the database.
      }
   }
}

This seems to offer the best of both worlds. I now require a Validator be passed in before I can save, and if the Validator is an interface, I can certainly allow clients to pass in the Validator they need. Of course, I could easily pass in a NOP validator.

Whatever…there are lots of different options, and each has their own set of tradeoffs. I’d like to avoid extensive debate surrounding which approach to save and validation is best and instead focus on the main point here, which is that achieving the right level of granularity for an entity can be very challenging. It requires more than just looking at the problem from a code level viewpoint and demands that we possess contextual information that will help us answer the question, “What’s the right level of granularity?”

Bring It Up a Level

The code above was a pretty simple example that illustrates the challenges of granularity at the method level. But the same challenges exist when developing classes, packages, modules and services. I discussed another, much higher level example, in the post on architecture all the way down. So what I really want to know is, “How do I determine the appropriate level of granularity?”

This is the million dollar question. Because granularity is a significant inhibitor to creating reusable software that’s easy to use. (Oh, and managing dependencies too.) If something does too much, it’s less reusable. If something does too little, it’s more difficult to reuse.

In the past, I’ve used the diagram at right (click to enlarge) to help illustrate one view of granularity. As can be seen, services are more coarse-grained than modules which are more coarse-grained than packages which in turn are slightly more coarse-grained than classes. This begins to help answer the question, “What’s the right level of granularity?”

If I’m concerned that a service I create is simply too coarse-grained and fails to maximize its reuse potential, I can break the behaviors of the service out into modules that are a bit finer grained and more reusable (of course, one might consider doing this in general). Then I can compose the services from the modules and reuse the modules across services. The result is different entities at different levels of granularity that lends tremendous flexibility to how I compose, use, and reuse software entities.

This provides some guidance on the level of granularity at which different types of software entities should be defined. However, it still doesn’t offer enough guidance to determine the right level of granularity for the save method in our example above.

Another Dimension

There’s another dimension that is relevant, and that ties in nicely (though in a rather subtle way) to the initial coding example. It has to do with the way we traditionally layer our software. The diagram at right illustrates this (click to enlarge). As we can see, as we move down the layered hierarchy, entities become finer grained.

While this isn’t absolute, and depends on your architectural principles and style, entities in higher level layers tend to be more coarse-grained than entities in lower level layers because they are an amalgamation of their own behavior and the behavior of entities in the lower level layers.

Armed with this information, we can determine the right level of granularity for the save method as long as we understand in which layer the code containing the save method lives. If it’s in the data access layer, and we place architectural constraints on business rules living in the data access layer, then we shouldn’t have any validation code that lives in the data access layer. If the save method lives in a class in the domain layer, however, it may be suitable for the save method to contain validation code in the domain layer and invoke another method in the data access layer that actually performs the save operation.

The Complete Picture

At this point we can make the following general statements:

Entities in higher level layers are more coarse-grained than entities in lower level layers.
Services are coarser than modules which are coarser than packages which are coarser than classes.

And if we combine these two ideas into a single thought, and overlay the two diagrams, we can begin to visualize the level of granularity for different types of software entities.

Certainly this doesn’t offer hard guidance. Realistically, there are very few architecture and design principles and patterns that are universally applicable. In fact, I’m not certain you’d ever create a “presentation service.” But certainly you might create a data service that is composed of multiple data access modules. And you might have separate data services that reuse a data access module. And you might have a business service that uses a data service and is also composed of multiple domain modules. But you definitely do not want a data access module that invokes a business service, nor a business service that references a presentation module.

So this general guidance can serve as useful information to help answer our question, “How do I determine the appropriate level of granularity for a software entity?” And it can also serve as guidance when establishing architectural principles and constraints that help determine where software entities with specific behaviors should reside. Because without some scheme that will help determine the appropriate level of granularity, it’s hopeless to imagine that we’ll be able to design reusable software entities that are also easy to use.

@kirkk.com blog: Complexity: Essence and Accidents

Mon, 19 Apr 2010 14:26:43 +0000

I’d like to take a short moment to offer an additional perspective to my discussion on OSGi: Complexity is NOT the Problem. I believe this perspective adds clarity to that previous discussion, as well. All initiated thanks to a tweet, which summed up the situation in much less than 140 characters.

So over the weekend, I turned to the essay that said tweet refers, and reviewed the essence and the accidents. From the article, I quote Mr. Brooks:

“…to see what rate of progress we can expect in software technology, let us examine its difficulties. Following Aristotle, I divide them into essence - the difficulties inherent in the nature of the software - and accidents - those difficulties that today attend its production but that are not inherent.”

This leads me, pretty clearly, to the following simple conclusion.

Today, development teams leveraging OSGi to build server-side applications are burdened by accidental complexity. As platforms and tools mature, the accidental complexity of OSGi will be reduced to near zero. Modularity though, enabled and enforced by OSGi, attacks the essence.

If the difference between essential and accidental complexity isn’t quite clear, I highly recommend you take a few moments and read Mr. Brook’s essay (linked above). And then, if you can find a copy of the book’s 2nd edition, take a look at Chapter 17, “No Silver Bullet” Refired (Note: If anyone can find the Refired essay online, please post the link in comments).

Throughout these two discussions, you’ll find subtle hints extolling the virtues of modularity. But only do this if you’re willing to exercise your brain with thought, because the connection you’ll discover might just be transformational!

@kirkk.com blog: OSGi: Complexity is NOT the Problem

Thu, 15 Apr 2010 18:39:31 +0000

The editors at SD Times have proclaimed that OSGi is too complex. Unfortunately, they miss the mark, and the following statement taken from the article is misleading.

We believe that OSGi, rather than simplifying server application development, has actually made it more complex, and there aren’t sufficient benefits to justify the added complexity.

This statement should have been tempered with context. It is not universally applicable, though is made to sound as if it were. OSGi may face challenges, but complexity of the OSGi framework is not one of the long-term inhibitors to adoption.

Technology Adoption Lifecycle

All technology goes through various phases of adoption. A lack of platform support and tooling may be inhibiting adoption today, but that is today’s problem, and one that is being dealt with. And there is a stark difference between complexity brought on by lack of tooling and platform support versus complexity associated with an ill-conceived idea. OSGi suffers today from lack of enterprise-class tooling and platform support. OSGi is not an ill-conceived idea.

Whenever an organization adopts a new technology, they’ll undoubtedly face some degree of initial complexity. New technology takes time to learn. As the innovators and early adopters continue to use OSGi, tooling is going to get better and developing modular applications using OSGi is going to get easier.

The question you must answer is whether the pain of adoption today is worth the gain of a modular architecture. There are many factors to consider when answering this question, but as the technology matures, the question will be easier to answer. As I’ve alluded to, other factors that have little to do with OSGi’s technical benefits will likely determine if it’s is able to cross the chasm.

Long Term Value

But the article’s biggest flaw is in stating:

And what’s the benefit again? Enterprise developers have written many, many server-side Java applications without using OSGi.

It’s important to understand its value over time. Let me pose some simple questions.

Is Spring too complex? It is far easier to create an application with concrete class references than it is to use dependency injection and abstractions.
Is Hibernate too complex? It is far easier to use JDBC than it is an ORM framework.
Is unit testing too costly? It is far easier to forego unit testing than it is to create a robust suite of tests.

Yet each contribute to more flexible software systems, and in that, there is significant benefit. So it is with OSGi and modularity, as well.

Leveraging OSGi to design modular applications will require us to learn new concepts, tools, and design paradigms, but there is no doubt that modularity has a tremendous upside. This makes the SD Times article paradoxical. OSGi does not increase complexity; the very benefit of modularity is that it helps us manage complexity.

@kirkk.com blog: OSGi - Feast or Famine?

Mon, 12 Apr 2010 16:49:54 +0000

I want to be careful here, but also very clear - I advocate OSGi and modularity, though am concerned about its future. In some circles, OSGi is hailed as a disruptive technology; in others, it lives in relative obscurity. There are some indications that adoption is growing, and possibly at some point, it’ll catch like wildfire. But right now, OSGi is the most widely acclaimed technology that nobody is using.

Maybe I’m too harsh. Or too impatient. Maybe I need to give it more time. I know some will argue that people are using OSGi, and they’re right. Some are. But of the more than 6 million Java developers worldwide, those using OSGi and designing modular applications is only a very small fraction.

So what’s the problem? What’s needed to drive adoption? How will OSGi and modularity succeed?

What’s In a Name?

One of the greatest inhibitors to adoption has been platform support. Until recently, few platforms exposed the virtues of OSGi to the enterprise developer. That’s beginning to change, and should help drive adoption. Another obstacle is tooling, which I discussed a while back when lamenting that one of the biggest barriers to adoption is no migration path. With better tools, we’re beginning to overcome that challenge.

But will it be enough? Will platform support and great tools help drive adoption? Sadly, possibly not. And part of the problem, I believe, has very little to do with the technology, and much more to do with its image.

So, what’s in a name?

OSGi is the dynamic module system for Java!

Modularity - Old and Stodgy

Recently, I commented on a recent conversation I had with a conference organizer and his reluctance to accept my speaking proposal on modularity. The general sentiment was that it’s an old, boring concept that can be traced back 40 years! He was interested in my talk on agility, but I explained that given the choice, I’d prefer to speak on modularity.

Currently, I’m participating in a similar conversation with the organizers of the Agile 2010 conference on a few sessions I’ve proposed on modularity. And now, Peter writes about a similar experience. This sends an important message to the OSGi community, and highlights an important challenge surrounding modularity. The message: Modularity is boring! The problem: The importance of modularity, and its support as a first class citizen on the Java platform, is not well communicated nor understood.

Not a Technology Problem

This is a serious problem that requires attention. In his post, Peter suggested that micro-services may be a more apt term for OSGi services, so as not to confuse the term with web services. He even hints at proposing a new JSR that would introduce the idea of micro-services as a new primitive on the Java platform. This is a good start, and micro-services isn’t a bad idea, but more needs to be done. The problem is not that OSGi isn’t a great technology with real benefits, the problem is that nobody cares about modularity.

Possibly it isn’t that they don’t care, but instead that modularity won’t sell. Nobody wants to buy a 40 year old technology, they want to buy the next great thing. And without question, developing modular software is not a new idea. But in general, the ideas surrounding SOA were not new either. Yet, without question, SOA has garnered significantly more attention than OSGi over the past decade. SOA has had its day in the sun; OSGi and modularity have not.

This is not a technology problem. The real problem is that OSGi doesn’t have a clearly articulated identity that explains why an organization must be using it, and organizations will adopt OSGi only if its adoption can be tied to real business benefits. Just like object-orientation in the 1990’s and SOA in the 2000’s, each were sold with a wonderful accompaniment of benefits that would reduce cost, speed delivery, model the real world, blah, blah, and blah. Whether they did or did not help organizations realize these benefits is arguable. Whether each were widely adopted is not!

Crossing the Chasm

OSGi has the potential to be this decade’s most disruptive technology. Unfortunately, the virtues of OSGi run the very real risk of never coming to fruition if it’s unable to cross the chasm. And sadly, it can never cross the chasm unless we find an effective way to sell, or reinvent, modularity.

Foremost, OSGi, modularity, and micro-services do not represent the paradigm shift that’s necessary. Too often, we talk about OSGi in terms of its technical merits alone. Unfortunately, that’s not enough. If OSGi is to cross the chasm, it is going to require something different. Something that people can really latch onto. Something that gets them excited. Modularity isn’t enough. Nor is plug-in architecture. Nor any other technically superior thing that OSGi is.

Naturally, this is only my perspective. My opinion. But OSGi has been a widely acclaimed technology for several years, and it has achieved very little enterprise adoption. While the move by SpringSource to donate dm Server to the Eclipse Foundation was hailed as a great contribution to open source, it’s also a clear indication that adoption of dm Server was gaining little traction.

Now, it is possible that an 800 pound gorilla could throw their weight behind OSGi and modularity that will help drive enterprise adoption. Maybe IBM’s move to expose the virtues of OSGi to the enterprise will be enough. With other vendors following suit, and up and coming vendors such as Paremus on the rise, that just might be enough. But if it is enough, I still believe the enterprise isn’t going to buy OSGi and modularity for the reasons we believe they should, they’ll buy something else very trendy in which OSGi and modularity plays a central role. Something new. Something fashionable. Because as I recall Ivar Jacobson alluding to at a keynote a couple of years ago,

The software industry is more driven by fashion than the fashion industry.

Sad. But true! And hopefully when it happens it won’t dilute the value of OSGi and modularity.

Ignore my Rant if You Want

OSGi is a great technology solution. But we also recognize that technologically superior solutions often fail to win. One of the most popular stories is of the video tape format wars. So my little pontification here should in no way be interpreted as questioning whether OSGi is capable, but instead whether OSGi will.

You can choose to ignore me if you’d like, scoff at me, or call this a rant. Maybe I’m just impatient. Either way, I’ll still continue to proclaim the virtues of OSGi and modularity. I believe in it…firmly! But the nagging feeling I have surrounding its ability to cement its place in the world of enterprise software development will continue to tug at me. And I wonder…will it ever succeed? And if so, how? And…when?

What do you think?

@kirkk.com blog: A Badge of Honor

Tue, 16 Mar 2010 16:09:16 +0000

Often times, I hear folks exclaim that they’ve been on “40 development projects over the past 10 years.” Or, “70 projects spanning a 20 year career.” They say this as if it’s some badge of honor. But I’m not so sure that it is.

Of course, there is value in widespread project exposure. Gaining experience with new technologies. Experiencing the dynamics of different teams. Understanding the challenges surrounding different organizational cultures. Each project is unique, bringing its own set of experiences, and these experiences are all incredibly valuable.

But when I see that someone has jumped from one project to another, it also leaves me wondering! Have they ever stuck around long enough on any single project to see their decisions through to the end? Have they ever had to live with the decisions they’ve made? An amazing compilation of knowledge is obtained by not only participating in the early stages of development, but also in maintaining the software system after it’s been released. To name just a few:

Dealing with the challenges of Phase 2, while also maintaining Phase 1.
Managing multiple branches of the software system. Merging those branches back together. Creating new branches.
A critical production issue arising about the same time you need to start heading out for that important customer demo.
Attempting to change a piece of code that hasn’t been touched in 10 months.
Watching the software system grow from nothing to a system that’s more than 500,000 lines of code. Keeping the build performant. Keeping the build working!
Being forced to live with seemingly meaningless early design decisions that haunt you over time.
Trying to upgrade versions of third party libraries while development is ongoing. Simply recognizing the right time to upgrade.

And there is so much more. Priorities tugging you in five different directions at one time. If you’ve had the luxury of living with a system (and the mistakes) you created, you’ll realize that there are very few, if any, decisions that shouldn’t be given conscious thought.

When an individual sticks with a project for a long time, they realize the importance of maintaining a clean design, a robust suite of unit tests, and how they package their software system. There is significant knowledge gained by sticking with a project for a long period of time. Perhaps the next time you hire a developer, it might be wise to ask them, “What’s the longest you’ve spent on any given project?” That may be more important than the number of projects they’ve been on.

Somewhere along the way, I picked up a small piece of advice that has helped serve as a valuable guide. There is a big difference between ten years of experience and one year of experience ten times. When you jump from project to project, never living with the decisions you’ve made, you have lost an opportunity to learn what works well and what does not.

@kirkk.com blog: Cost: Mac or PC? A Look at TCO!

Thu, 11 Mar 2010 16:38:18 +0000

I’ve owned Macs for about four years now. I use them daily. One of the biggest complaints I hear surrounding Macs is that they’re more expensive than their PC counterparts. This recent article in CIO examining total cost of ownership (TCO) for businesses using Macs left me wondering. For personal use, are Macs really that much more expensive than PCs, especially when factoring in TCO? Unfortunately, it’s not so black and white. Let’s look at some numbers.

For this little exercise, I’ve decided to compare sample TCO over a period of time (ie. 2 years) for a Mac and a PC. This includes the initial purchase cost, ongoing support costs, and software purchases. It does not include any personal time that you would need to devote to maintaining and managing the device, such as reinstalling the operating system, cleaning up unused applications, dealing with anti-virus software and configuration, and general troubleshooting tasks.

I’ve made my most valiant effort to present an unbiased view and have arguably given favor to the PC. It would have been easy to close the price gap much more quickly if I were to toss in a few not-so-easy to measure items, such as the likelihood that you’ll have to take your PC to a professional to have it saved. But I’m sticking with hard numbers; the stuff that’s easy to measure. Let’s get started.

The Initial Purchase

I picked two models, fairly equal in their capabilities - a Dell Inspiron 15 and a 15″ MacBook Pro. When purchasing the Dell, I had to configure it so it had specs equal to the Mac. Let’s look at purchasing and configuring the Mac first.

The MacBook Pro

The 15″ MacBook Pro we’re going to purchase comes already loaded, so we aren’t going to make many customizations to it. We’re going to go with the entry level model, which prices out at $1699. The general specs include a 2.53 GHz Intel Core 2 duo processor with 4 GB of RAM. All we’re going to do is increase the hard drive to 320 GB, which adds $50 to the price, and purchase iWork, which adds $49 (Pages, Numbers, and Keynote). For those not familiar with iWork, this is what we’ll use on the Mac instead of Microsoft Office.

Total price for the MacBook Pro: $1798.00.

The Dell Inspiron 15

Keep in mind that the Inspiron is under Dell’s “Home” category. What this generally means is that it comes pre-loaded with a whole bunch of useless trial software that expires in either 30, 60, or 90 days. After that initial trial period, you’ll have to purchase a license to continue using it. For a business, you’d probably purchase the Latitude to avoid all of this unnecessary garbage. To keep the price down, I’m going to go with the Inspiron 15. See…told you I was giving favor to the PC!

To get a somewhat even comparison, I’m going to have to customize this thing to bring it up to equal specs as that of the Mac we’re going to purchase. The initial cost of the base machine is $499. But without any further customizations, that machine doesn’t offer much horsepower. First, I’ve got to add a Core 2 duo processor clocking in at 2.53 GHz. This adds $225 to the price. We do need the wireless -N card, which adds $40. Why would this not be standard? Odd!

Next is Microsoft Office Home and Student 2007, which adds $119. We need this if we want to create spreadsheets, documents, or presentations. I recognize I could have gone with OpenOffice, but I find that most people choose Microsoft Office, so that’s what I’m doing here. Since we’re running Windows, we need anti-virus software. With the Dell model, there’s an option to purchase a 36 month license of McAfee. I’m going to jump on that because it only adds $40 to the price.

At this point, we’re finished with the Dell. There are a few things we could have done to even out the playing field a bit more. For an additional $45, we could have upgraded to the 9-cell battery that would have given us up to 8 hours of battery life. Keep in mind the Mac has a battery that gives us about 7 hours. Additionally, the version of Microsoft Office we purchased doesn’t come with Outlook, so we’ll still need to find an e-mail client once we get our laptop. But I just couldn’t see spending another $160 to purchase the Small Business edition that comes equipped with Outlook. The Mac will have Mail.app preinstalled for us, which is a nice e-mail client with Exchange integration. Already, the cost of software licensing is a sign of things to come that’s going to factor heavily into TCO.

Total price for the Dell: $883.

The Software

While these are each good machines, to get great things done requires software. Each of us have different needs surrounding the software we use, so I’ll offer up a few samples based on what I use. It won’t take long to get a clear understanding of the significant impact this has on TCO. Choose your own software stack, and see how it adds up. Then, let me know if you’re findings are consistent with mine.

First, I want to purchase software that allows me to create some nice drawings. For the Mac, I’m going to purchase OmniGraffle, which comes in two versions, standard ($99.95) and professional ($199.95). For Windows, Visio is a good option, which also comes in two different versions, standard ($259.95) and professional ($559.95). Let’s go the cheaper route on each. This brings the total price for the Mac to $1897.95 and the price of the Dell to $1082.95. Note: If I had purchased the professional version of each, the TCO at this point would be 1997.95 and 1382.95, respectively.

Next, I want some screen capture software so I can create some short training videos for online publication. For this, we’ll use Camtasia. The Mac edition of Camtasia prices out at $99.00, and the Windows version comes in at a rather lofty $299.00. TCO for the Mac is now at $1996.95 and the Dell is at $1381.95.

See a pattern developing here yet?

Keeping this post at a reasonable length, I’m only going to make one more purchase. At some point throughout my time as owner of one of these wonderful products, it’s likely that a new version of an operating system is going to arrive. And I’m going to want to upgrade. When Snow Leopard for Mac hit the market, the cost to upgrade was a paltry $9.95. Alternatively, the price to upgrade to Windows 7 Home Premium comes in around $119.95. In fact, a full license of Snow Leopard is only $29.95. A full license of Windows 7 Home Premium? $199.99! Again, the pattern is developing. TCO for the Mac is now $2006.90, while the Dell is at $1501.90.

Additional Analysis

At this point, the Mac is still slightly more expensive. But an initial price difference of $1200 was quickly dwindled down to just over $500 by the time we were finished. Clearly, the stark difference in the initial price is quite different from TCO. I could easily keep going, purchasing software that I want and need, and each time find that the Mac edition of the software comes in priced far below the Windows version. But I think I’ve illustrated the pattern here. The cost of software for Windows is more expensive than corresponding software for the Mac, and throughout the lifetime of ownership, TCO for Windows will approach, if not exceed, that of the Mac. For instance, Apple offers the option of purchasing a 5-license family pack of iWork for a mere $20 more than a single copy. A 5-license copy of Office is going to be quite expensive ($595).

Additionally, I have not broached the subject surrounding the time you’ll spend maintaining the Windows machine to keep it running smoothly. In four years on a Mac, I haven’t devoted any effort to tuning the machine for performance. It just works. After about six months on a Windows machine, I find that I’m always tinkering with it to maintain optimal performance. Since I can’t put an accurate price tag on this, I haven’t discussed it. But it is significant. Ok, I’m a little biased here, aren’t I?

Finally, I could have gone with the 13″ MacBook Pro with the exact same specs (just a smaller display) as the Inspiron 15 for an entry price of $1448.00. And the 13″ Mac is a great machine. After purchasing the software I want, the price for the 13″ Mac comes to 1656.90. That’s only $155.00 more for the Mac!

The Clear Winner Is…

The winner…you decide! All too often we compare Mac and PCs based on initial cost, failing to factor in the longer term TCO. When comparing Apples-to-Apples (errr…I mean PCs) though, the difference isn’t as stark. If you want a low end machine with a short lifespan, Dell (and many other PC manufacturers using Windows) offer this, including their line of netbooks.

It is possible to purchase a Dell for under $500. But remember, you get what you pay for. And side-by-side, machines of equal horsepower and capability are going to result in similarly equal TCO. By the time you spec out the machines evenly, and factor in TCO, the price of a Mac is arguably equal too, if not less than, that of a Windows PC. Either way, I know what my choice is!

@kirkk.com blog: Agile Animations - Big Teams

Mon, 08 Mar 2010 18:33:58 +0000

@kirkk.com blog: Architecture Lives Here

Thu, 04 Mar 2010 16:48:21 +0000

Last week at OSGi DevCon, I attended Peter Krien’s Advanced OSGi tutorial. The tutorial focused on OSGi services, and I had just come off a rather lengthy e-mail discussion with Peter on a similar topic. So this was pretty good timing. I want to share with you one aspect of the discussion, and its relationship to architecture. It’s quite simple, though makes an important point.

In the past, I’ve talked about the importance of designing the joints of a system. OSGi services represent these joints, since services are essentially a module’s published interface. In general, if the implementation details of a module are well encapsulated, then implementation details can change without impacting the rest of the system. In other words, we minimize that nasty ripple affect of change if we confine change to a single module. But changes that span modules is nasty. Let’s take a few visuals to illustrate the point here.

Where’s the Architecture

Commonly, we visualize the module structure of a software system using a simple diagram similar to what’s shown at right (click to enlarge). We see the modules and the relationships between them. Of course, the relationships between modules represent the joints. Pretty simple stuff! Unfortunately, our attention is immediately drawn to the modules themselves, but not so much the joints.

And we really must be much more concerned about the joints than we are the implementation details because changes in the joints of the system are more complex and costly. This is where we need flexibility and it’s also where we need stability. Emphasizing module implementation is important, but flexibility in the joints is more important.

Illustrating the Joints

In the session, Peter annotated the diagram using a new convention, which I’ve depicted in the diagram at right (click to enlarge). The red triangles more clearly illustrate the joints. Here, our attention is drawn more toward the design constructs that span module boundaries. No longer is emphasis placed on the implementation details, but instead has moved to focus on the relationships between the modules. And the direction of the red triangle also helps illustrate the direction of the dependency relationship between the modules. In other words, these red triangles are the services.

The Real Architecture

Now, because the module itself represents behavior that is encapsulated, we can remove the modules from the diagram altogether and show only the joints, as shown at right (click to enlarge). This is where the impact and cost of change is most significant. And this is where we really need to ensure we have the greatest flexibility. These are the decisions that are most architecturally significant.

A Short Digression

Now I don’t think it’s necessarily important to start drawing architecture diagrams this way. In fact, I’ve never felt that it’s critically important to draw these pretty diagrams in the first place. While diagrams (and documentation) can be useful, they are little more than a pimple on software development’s ass. There are much more important things to worry about than diagrams. But I digress, and in general, this simple exercise helps recognize the importance of designing the joints of the software system.

The Real Point

Of course, while this example illustrates the point using modules, it’s not just the joints between modules that are important, but the joints at many different levels of the system, including applications, services, packages, and classes. In other words, the point where two entities connect is where the real architecture lives.

@kirkk.com blog: OSGi DevCon Slides

Fri, 26 Feb 2010 16:09:05 +0000

After the OSGi DevCon keynote, I had a few folks stop by and ask me for the slides. They are now available on slideshare. Usually, when browsing the slides without discussion, the lack of context can cause some confusion. So if you have any questions regarding the content, please feel free to reach out. Enjoy!

OSGi in the Enterprise: Agility, Modularity, and Architecture’s Paradox

View more presentations from pragkirk.

@kirkk.com blog: OSGi DevCon Preview

Wed, 17 Feb 2010 16:24:21 +0000

I’ll be at OSGi DevCon next week, having been given the opportunity to present the keynote session on Tuesday. The conference is co-hosted with JAX London. I thought I’d take a moment before jumping on the flight to offer a preview of what I’ll be speaking about.

The Cost of Complexity

The title of the session is OSGi in the Enterprise - Agility, Modularity, and Architecture’s Paradox. We all know that OSGi isn’t a new technology. It’s been around for over a decade. But the recent surge in interest in modularity on the Java platform is overdue. The complexity of software is increasing exponentially. Did you know:

In 1990, there were 120 billion lines of code
in 2000, there were 250 billion lines of code
The number of lines of code doubles every 7 years
50% of development time is spent understanding code
90% of software cost is maintenance and evolution

Let’s put this in perspective using the visual at right (click to enlarge), where we can see that the amount of code doubles every seven years. Pretty staggering! But let’s put this in perspective for what it means over the course of the next seven years, as well. It means that between 2010 and 2017, we’ll produce more code than the total amount of code ever written combined! That’s staggering!

I decided to do a bit of research to see if I could find examples that supported these claims. I did:

Since the Spring framework was released in 2002, the number of lines of code has grown more than 500% through the release of Spring framework 2.5 in 2008.
FreeBSD contained roughly 8 million lines of code in 2002. In 2009, it was close to 16 million.
In 2004, the Linux Kernel contained approximately 6 million lines of code. In 2009, it was around 12 million.

I’m sure many of us have experienced this phenomenon. Systems tend not to get smaller over time. And we also recognize that in almost every way, larger software systems are inherently more difficult to design, build, manage, and maintain than are smaller software systems.

Yet, such phenomenal growth is desirable. We can only hope that the systems we create are in such high demand that we have the opportunity to grow them over time. A software system that doesn’t change, dies. Evolution is a big deal! As Lehman’s law suggests:

As a system evolves, its complexity increases unless work is done to maintain or reduce it.

Add it all up, and there are some key takeaways here. We need something that will help us understand complex systems. We need something that help manage the complexity. We need something that will help ease maintenance. We need something that will help us deal with the natural evolution of software systems. We need something that will allow us deal with the natural architectural shifts that occur as a system grows to accommodate demand. For a long time, a central ingredient has been missing. But not for much longer, because the enterprise will get its OSGi!

For the rest of the story, see you in London.

Source of statistical information above: http://users.jyu.fi/~koskinen/smcosts.htm

@kirkk.com blog: Agile: The New Era

Wed, 10 Feb 2010 16:07:21 +0000

It’s housecleaning time again, and like last time, I stumbled across an article I wrote back in 2006 that I don’t believe ever reached publication (at least, I don’t think it did…how am I expected to remember what I did in 2006?). For the most part, I’ve left it in its original state, except that I removed the Agile Manifesto and 12 supporting principles. There are easily enough found on the Agile Manifesto website, and the article is long enough without this duplication. The wordle at right shows the most common words used in this document (click to enlarge). Here, in it’s otherwise unadulterated glory, is Agile: A New Era of Software Development.

Agile: A New Era of Software Development

Embrace Change

Writing code is easy, but developing software is hard. While syntax errors are common, their severity pales in comparison to the logic flaws inherent in many software systems. The difficulty in software development is not writing code or applying a certain technology stack. Instead, the challenge lies in the specification and design of the software itself. Therein lies the essential complexity of software development, an idea introduced by Frederick Brooks in his 1987 article titled, “No Silver Bullet” [Brooks]. The most difficult aspect of software development is deciding what, exactly, needs to be built.

There is certainly evidence backing this claim. The original Chaos Report shows the top three impediments to a successful development effort are lack of user input, incomplete requirements and specifications, and changing requirements and specifications [CHAOS]. No other activity, if done incorrectly, stands to compromise the system more than incorrect requirement specifications.

It might not be so difficult were software a concrete entity, existing in a world where we could easily visualize it’s structure and behavior, allowing us to more reliably assess and share the impact of change. But software is a highly conceptual, invisible construct. It is considered infinitely malleable by those not intimately familiar with the conceptual complexity of it’s structure and behavior. The contractor building your home would look at you with incredulous disbelief if you suggested that the house he has 90% complete no longer met your needs, and you asked that he move walls. Or imagine how ridiculous it would sound to suggest that a new third floor be inserted to a 100 story skyscraper. Physicists labor on with firm belief that there exist an underlying set of unifying principles to serve as guidance. Or at least, there are laws of physics that we hold to be true. There are no such rules or principles that guide software development. We are left with the imagination and dreams of our clients, and they demand and deserve rapid response to change.

We have made valiant attempts at conformity. Ceremonial processes attempting to define standardized activities that guide the development process have failed, however. We cannot define detailed up-front requirements specifications and expect them to survive the development lifecycle intact. We cannot establish an initial design of the conceptual construct and expect the structure to go unscathed throughout the process of construction. Software development is an error prone human activity involving experts with varying backgrounds and skills who must come together and attempt to communicate uniformly, working as a team toward a common goal. While tools and process do help, we must also accept that change is expected. We cannot treat change as evil. Instead, the tools and process used must allow us to accommodate change, treating it as an inherent part of software development. Changing requirements is a rule of our game. The software we develop must be malleable and adaptive to change, and the process we use must embrace change.

We often draw comparisons between software development and various manufacturing processes. As Larman points out, however, manufacturing is a predictive process [Larman]. Herein lies one of the greatest differences between software development and the manufacturing processes to which we often draw comparisons. Manufacturing is a repeatable activity, with high rates of near-identical creation where a consistent product is produced in assembly line fashion. Little change is expected, making it possible to reliably estimate and predict the outcome. Software development is much more like new product development, where evolutionary specifications and adaptive planning is necessary to deal with the many unknowns that lie ahead.

Agile Principles

In early 2001, a small group of industry experts converged in Utah to discuss alternatives to heavy, document driven software development methods. Emerging from this meeting was the Agile Manifesto, a symbolic proclamation endorsing the virtues of a lighter, more flexible, people-oriented approach to software development, giving birth to the agile software development movement. (Since this is already a long article, I’ve snipped the manifesto and principles, which were included in the original version. If you’re interested, you can view the manifesto and its 12 principles on the Agile Manifesto website.)

<Snipped the Manifesto for Agile Software Development and 12 Principles>

The ideas behind these 12 principles are simple, and contain no hidden messages. Of course, there are different techniques embodied within various agile processes that support these principles. The one certainty is that agile teams definitely work differently from their less agile peers. They recognize there is one end goal - to create a working, functional software product. With that in mind, they work very closely with project stakeholders throughout the development lifecycle, knowing it is the stakeholders who possess the knowledge the system must embody. Agile teams work very hard to deliver working software iteratively and incrementally, and they adopt techniques representative of that ideal.

Agile project managers tend to favor intense communication and collaboration over heavy documentation. Empowering team members to make decisions enables responsiveness to change. Facilitating and negotiating requirements scope provides important feedback, helping plan future iterations, where each iteration produces a deliverable that can be shared with clients and stakeholders. Instead of forcing the team to follow a predictive project plan, agile project managers are more opportunistic. They prioritize features based on stakeholder feedback, and make adjustments as the iterations progress. Concurrent and parallel activities are favored over a phased approach. Agile project managers tend to guide the team instead of manage the team, and strongly discourage unnecessary overhead.

Agile developers work with a similar set of goals, knowing functional software must be delivered early and often. They work judiciously to grow a code base built upon a solid foundation, where each day represents a step forward. They integrate frequently, and do not tolerate failed builds. A rich suite of tests provide the courage necessary to respond to change when the need arises. They avoid the notion of code ownership, empowering other developers to make improvements to any component of the software product.

A common misconception is that agile processes discourage all documentation. This is untrue. Agile processes discourage unnecessary documentation, favoring collaboration as the preferred technique. Instead of using documentation to drive communication, agile processes favor face-to-face communication. Documents are encouraged by agile processes, so long as the need is immediate and significant.

Transitioning to Agile

Agile software development is based upon the fundamental premise that we must drive and respond to change quickly. The Agile Manifesto and 12 supporting principles serve this premise well. Advocates of agility claim speedier delivery of software, software with more business value, increased team productivity, higher quality systems, and a more enjoyable development experience. I believe each of these to hold true. Agile teams not only welcome change, they are able to respond to change at all levels of development. A project manager might discuss a changing requirement with a business client, empower a business analyst to schedule a meeting with the client to discuss further details, while a developer assesses the impact of change knowing she has the courage to accommodate the request because of the rich suite of unit tests in place.

Saying you’ll be more responsive to change and creating an environment that embraces change are separate beasts, however. Practicing agility is hard work, especially if your team is accustomed to more traditional approaches. As with many things new and unfamiliar, some resistance will no doubt arise by those who aren’t fully convinced. Agile projects differ greatly from their less agile counterparts, and skeptics will have many opportunities to express their discontent. As someone experimenting with agility, you may even have doubts. But don’t be discouraged, and give your agile transition the time it deserves.

One of the most significant changes you may experience is a feeling that you’ve been thrust into a chaotic nightmare. I doubt it’s unusual to feel this way. You’ve lost the security of detailed requirements specification and user sign-off. You are writing code without the comfort of knowing exactly what your stakeholders want. The detailed plans that have served as your security blanket on past projects no longer exist. And the celebrations accompanying completion of your various phase milestones are gone. Of course, these were all false comforts anyway. Stakeholders always changed their minds. Your detailed requirements and plans were outdated as quickly as they were completed.

Instead, you’re now working in shorter iterations with vague requirements. Initial releases early in the lifecycle may be completely thrown away. Your first few weeks may seem wasted, with little or no valuable artifacts produced. Naysayers will immediately come forward and cite the lack of progress. Previously, those first few weeks or months were spent producing very detailed requirement specifications and beautiful design models. But don’t give up yet. In that previous world, you were only delaying risk and postponing integration, avoiding the most difficult aspect of software development until the end of the lifecycle. Now you’re attacking risk early, prioritizing features, and working hard to develop a functional piece of software as early as possible. Progress may not be at breakneck speeds, but you are learning a tremendous amount about the requirements of the system, and your velocity is sustainable. Additionally, you are also performing a number of other important lifecycle activities.

Depending on the level of ceremony and bureaucracy within your organization, you will experience varying degrees of success when adopting agile techniques. As with any new technology adoption, it’s best to phase the transition. Some agile techniques are easier to adopt than others, and some serve as valuable catalysts to adopting additional techniques in the future. Don’t attempt to completely redefine how you work. It’s relatively easy to phase the agile transition, and you’ll want to adopt those principles that offer you the greatest initial reward.

For instance, if you’re struggling to produce quality software at a consistent rate, implementing a continuous integration strategy will help you frequently verify the quality of your work. In addition to the comfort of knowing you have a product always in a functional state, the ability to share the product with clients using functional demos and prototypes will tighten the feedback loop and offer valuable insight to the client’s perception of the software. In a number of cases, I’ve found this to be valuable in identifying subtle requirements that can be difficult to identify in other requirements elicitation venues.

Empirical Evidence

In recent years, there has been a significant amount of research comparing agile development methods to their waterfall counterpart. In Agile and Iterative Development: A Manager’s Guide, Craig Larman illustrates the advantage of agile development through detailed analysis of multiple studies[Larman]. The compilation of his results are illustrated below.

A study by Alan MacCormack at Harvard Business School explored whether evolutionary development techniques yielded better results than the waterfall model. The study included applications ranging from application software to embedded systems, with median values of nine developers spanning a 14 month development cycle.

A key conclusion of the study, in which 75% of participants used agile techniques compared to 25% using waterfall, explained releasing software earlier, rather than later, contributed to a lower defect rate and higher productivity. There was little evidence showing that a detailed design specification resulted in a lower defect rate, however, reviews with peers did help in reducing the rate of defects. The study found that iterative and agile practices have a significant impact on defect and productivity factors, as indicated by the following points.

Releasing a system with 20% of the functionality complete is associated with a decrease in the defect rate of 10 defects per month per million lines of code as compared to waiting to release a product until 40% of the functionality is complete, and an increase in productivity of eight more lines of source code per person-day.
Continuous Integration, the idea of integrating and testing code as it is released to your source code repository, resulted in a decrease in the defect rate of 13 defects per month per million lines of code, and an increase in productivity of 17 lines of source code per person-day.

The study also found four practices that were consistently used by the most successful development teams. The first two are deeply embedded in the ideals of agile software development.

Releasing early and often to project stakeholders, using an iterative lifecycle.
Continuous integration, with daily builds including regression testing.
Teams with broad experience delivering multiple projects.
Careful attention to modular and loosely coupled, componentized architectures.

In a separate study [Shine], 88% of organizations cited improved productivity when using agile methods, and 84% cited improved productivity. 49% stated that the cost of development was less when using agile methods. Additionally, 83% claimed increased business satisfaction and 26% claimed significantly better satisfaction. Another study by Boehm and Papaccio [Boehm] discovered that a typical project experiences a 25% change in requirements, while yet another [Johnson] showed that 45% of features were never used.

There have also been many research efforts devoted exclusively to the analysis of waterfall methods. Below is a summary of these findings, taken from a variety of studies.

Scope management related to detailed up-front requirements was a significant contributing factor of failure [Thomas].
The U.S. Department of Defense (DoD), when following a waterfall lifecycle, experienced a 75% failure rate [Jarzombek]. This resulted in the DoD adopting a more iterative and agile approach.
On a study including 400 waterfall projects, only 10% of the code was deployed. Only 20% of code deployed was used. The main factors included changing and misunderstood requirements [Cohen].

As these studies clearly illustrate, there is significant evidence showing that agile and iterative techniques offer significant advantages over the waterfall model of development. In fact, for larger projects, the statistics supporting agility were even more pronounced.

Conclusion

There are a variety of agile processes available to choose from, and each abide by the spirit of the manifesto and it’s 12 supporting principles. The agile movement and it’s supporters recognize that software development is a human (though not always humane) activity. Instead of forcing process on people, agile methods allow process conformance to people. Good people, working toward a common goal, can achieve great things will little ceremonial process, assuming you give them an environment that empowers them. Solid empirical evidence backs this claim. And if the quality of people is in question, it’s doubtful that any process will produce success.

References

[Alliance]. The Agile Alliance. Manifesto for Agile Software Development. 2001. http://www.agilemanifesto.org
[Boehm]. Boehm, B, and Papaccio, P. Understanding and Controlling Software Costs. IEEE Transaction on Software Engineering. October 1988.
[Brooks]. Brooks, Frederick. No Silver Bullet: Essence and Accidents of Software Engineering. 1987.
[CHAOS]. The Standish Group International, Inc. The CHAOS Report. 1995.
[Cohen]. Cohen, D., Larson, G., and Ware, B. Improving Software Investments through Requirements Validation. IEEE 26th Software Engineering Workshop. 2001.
[Jarzombek]. Jarzombek, J. The 5th Annual JAWS S3 Proceedings. 1999.
[Johnson]. Johnson, J. Keynote speech, XP 2002, Sardinia, Italy. 2002.
[Larman]. Larman, Craig. Agile and Iterative Development: A Manager’s Guide. Addison-Wesley, 2004.
[MacCormack]. MacCormack, A. Product-Development Practices That Work. MIT Sloan Management Review. 2001.
[Shine]. Corporate Report. Agile Methodologies Survey Results. Shine Technologies Pty Ltd. Victoria, Australia. 2003.
[Thomas]. Thomas, M. IT Projects Sink or Swim. British Computer Society Review. 2001.

@kirkk.com blog: Small Things Matter

Wed, 03 Feb 2010 15:55:40 +0000

Story of the Concorde

On July 25th, 2000, flight 4590 crashed. It was the first, and only, crash of the famed Concorde. Eventually, it would lead to retirement for the amazing aircraft. Investigators spent countless hours poring over the wreckage, and placed blame on a piece of runway debris that slashed the tire. A piece of that tire struck one of the fuel tanks, causing it to rupture and the plane caught fire. Case closed, right? Not so fast.

Surely a small piece of runway debris shouldn’t bring down a commercial airliner! As it turns out, there is quite a bit of contention among experts surrounding other factors that may have contributed to the crash. Some argue that it was a complex chain of events, all coming together, that brought down the aircraft. The plane was missing a spacer between the two wheels, had too much fuel in the tank, attempted to takeoff in unstable conditions, and was overweight. The flight was also delayed, causing angst among the flight crew. Finally, a required daily runway inspection was not performed.

Perhaps, if the runway inspection had been performed, the piece of debris would have been spotted. Or had the aircraft not been overfueled, the piece of tire may not have caused an increase in fuel tank pressure that some say caused the tank to burst. Had the spacer been installed, it’s possible the tire would have never burst. A series of small events, each contributing in their own way, to the fatal crash.

No Matter How Small

The story reminds me of the importance of attention to detail in software development. Of how the aggregate of all of the small, seemingly insignificant decisions we make on a continuous basis can have long-term consequences on the future of the software system. Possibly even your organization. Every time you design a class, define a variable, write a test, create a package, build a module, modify a method, or make a design decision, you are affecting the future of your system in some unsuspecting way.

What may seem insignificant today, can have a detrimental affect tomorrow. Really, there are no small, insignificant decisions in software development. I’m reminded of how important it is to make conscious decisions that are given careful thought, no matter how small. It also reminded me of a poem on build automation and continuous integration that I read a while back on the Test Early blog.

FOR WANT OF A BUILD

For want of a build, a test case was not executed
For want of test case, a defect was not detected
For want of a defect report, a bad release was promoted
For want of a good release, a strategic customer was lost
For want of a customer, a development team was reduced
For want of developers, a product stagnated
For want of a product, a company was lost
And all for the want of a build…

@kirkk.com blog: 3 Pillars of Architecture

Wed, 27 Jan 2010 16:42:39 +0000

I’ve spent some time thinking a bit more deeply about a few of my recent posts on software architecture, and have come to the following revelation.

Architects architect architecture!

Don’t let the triviality of this statement undermine its depth. While each of the three words are variations of the same thing, each have different contextual meaning. Let’s tease the statement apart to see what I mean.

Architects - Humans create software architecture, and for architecture to be effective, we must also be able to understand the architecture. In Eliminate Architecture, I cited a definition of architecture that introduces the social dimension. Architects signify the social pillar.
Architect - The way we arrive at architecture is through some process or series of steps. We might create diagrams or software architecture documents. We might write a little code (proofs, spikes, prototypes) to determine the viability of architecture. There are many different activities we perform as we create the architecture. Architect signifies the process pillar.
Architecture - In Modularity and Architecture, I offered up a few industry definitions of architecture. Common keywords that span definitions include components, composition, interfaces, subsystems, and structure. Architecture signifies the technology pillar.

To ensure balance, we must give attention to each of the three pillars. Additionally, each pillar is related to the other. For instance, ignoring the social pillar impacts the other two in unexpected ways.

The Social Pillar

Turtles and Architecture generally focused on the social pillar of software architecture, but also talked a bit about how the technology pillar can increase understanding, visibility, and transparency. The general sentiment is that architects must focus on more than just concepts and developers must focus on more than just code. There is a middle ground that demands attention, as well.

I used a visual similar to what’s at right (click to enlarge) to illustrate the middle ground. It also illustrates how aspects of the technology pillar can help increase understanding and transparency of architecture. Increased understanding of the architecture hopefully leads to improved structural quality (technology pillar) and transparency of the process (process pillar).

The Technology Pillar

Architecture All the Way Down primarily focused on the technology pillar. The visual at right (click to enlarge) illustrates this view. Again, we see the huge gap that exists if we fail to emphasize the importance of modularity. The rightmost portion illustrates how modularity fills the gap - architecture all the way down. Of course, other gaps are also created if we ignore any of the other aspects, such as class design. Note that as we move from services to modules to packages and classes, we increase the granularity along the way. Our classes should not be as fine-grained as our modules, nor our modules as fine-grained as our services.

Additionally, each entity solves a different type of problem (or provides a different type of advantage), as illustrated by the bars at the bottom. All are units of composition, but only services and modules are units of deployment. Services are reused through distributed computing, whereas module reuse is constrained by process boundaries. The technology pillar affects the other pillars. An inflexible rigid architecture makes it difficult for people to understand and communicate (social pillar) and inhibits how quickly we’re able to respond to change (process pillar).

The Process Pillar

The process pillar is one that I’ve not spent much time discussing. Certainly, it’s important though, and includes various aspects like deferring commitment to significant architectural decisions, evolutionary and emergent architecture, and reversibility. The visual at right illustrates the process pillar (click to enlarge). It’s not as descriptive as the other diagrams, I admit. Anyone have something better that illustrates the process pillar?

I did talk a little about these ideas in Agile Architecture, Lean Principles. But certainly more needs to be fleshed out surrounding the process pillar. This tends to be where most spend their time when discussing agile architecture, but the other pillars are certainly important. The process pillar affects the other pillars. A bad process accompanied by bad practices results in an inflexible architecture (technology pillar) that noone is able to understand (social pillar).

@kirkk.com blog: Update on Java Modularity

Fri, 22 Jan 2010 15:43:05 +0000

For those that follow this blog, but not the APS blog, I’ve just published a synopsis of modularity on the Java platform. With a lot of momentum continuing to build around OSGi, we’re still in a state of limbo surrounding a standard module system on the Java platform. Some might argue that OSGi has already won - it is the defacto standard module system. Possibly.

Oracle has offered us very little direction on their future plans. There was little fanfare surrounding WebLogic dm Server, with only sparse references and no official word from Oracle. I’m hopeful we can expect to hear some news of the direction they intend to take in their upcoming webcast, though I’m not expecting them to make any significant statements at such a fine level of technical detail. Coincidentally, Apple is holding their own event that very same day, and is expected to announce their new tablet. Let your imagination run wild. Those two events should make for a very exciting day.

Regardless, I’ve had quite a few conversations over the past several months surrounding modularity and OSGi. I have my suspicions on Oracle’s intent, though am reticent to share at this time. Read the APS post titled, Java Modularity - Time to Set Sail.

@kirkk.com blog: Developer, You Have a Voice

Fri, 15 Jan 2010 16:39:25 +0000

I’ve been thinking a bit more about the list of disruptive technologies on Richard’s list, and then I watched Dan Pink’s TED talk on the surprising science of motivation. There must be some relationship between the list, which is comprised of almost all open source software products, and Dan’s assertion that the 20th century reward system won’t work for the cognitive tasks performed by workers in the 21st century. What is it, though?

We develop open source software to scratch a personal itch, ease the pain in performing a certain type of task, or create a more compelling alternative to a commercial product. While the professional open source model has emerged the last few years as a way for the open source community to create a sustainable business model, none of the open source technologies on the list were initially developed that way. They were developed in response to need. Ant because Java lacked a good build system. Spring because Java EE was cumbersome and bloated. JUnit to help increase quality. In many cases, these tools have grown to become defacto standard technologies widely used by enterprise development teams.

In Dan’s mind, the new incentive program within organizations must revolve around three things - autonomy, opportunity, and purpose. We must be given autonomy, or the empowerment to make our own decisions. We must be given the opportunity to master something that matters to us. And we must be given purpose, which is the desire to do what we do in the service of something larger than ourselves. Dan notes that financial incentive is also important, but is not the decisive factor in what motivates us.

There are a lot of ways to connect the dots between the open source products so prevalent on Richard’s list and Dan’s point about incentives. I’ll allow you the opportunity to connect these dots any way you wish. A few that immediately come to mind for me include the way commercial software is sold, corporate incentive programs, empowering developers, and flaws in corporate culture. But here’s something else to chew on.

In most of the cases, it was the developer who spurred adoption of the most disruptive application development technologies of the last decade. We weren’t motivated to develop or adopt these great products because of financial incentive or a reward system. Sure, in some cases that’s a positive side affect, but it is not the force that motivates. Instead, we were motivated because they make our jobs a little bit easier and our software a little bit better.

Developers are fighting like hell to create better software. While a lot of commercial vendors are selling shelfware (they aren’t selling it to the developer, mind you), developers are driving adoption of the technologies that are making a difference. Developers seek autonomy, opportunity, and purpose. Given corporate culture, it’s not always easy to find. But if the last decade is any indication, we are finding it, developers do have a voice, and that voice is being heard.

@kirkk.com blog: Fun With Modules

Tue, 12 Jan 2010 00:00:33 +0000

Tight coupling between modules is a bad idea, and the worst form of coupling is cyclic dependencies between modules. Fortunately, there are a few techniques we can use to break the cycles. They are Callback, Escalation, and Demotion, and I’m going to walk through some examples that show each of them in action.

Then, once the dependencies are broken, we’ll look at two more techniques that allow us to invert and eliminate the relationship altogether. The code for each of the samples can be found in the edcie project on my Google code repository. Each example includes a build script and a simple test case. To execute them though, you’ll need GraphViz if you want to use JarAnalyzer. To invoke the build scripts without invoking JarAnalyzer, you can simply type:

ant compile

Keep in mind that each variation of the system has the exact same behavior!

The Example

The example we’re going to use to drive the remainder of our discussion is incredibly simple. We’ve got a Customer and a Bill class that we’re going to bundle into two separate modules - cust.jar and bill.jar. There’s also a test case called PaymentTest that serves as a sample client to drive the interactions between the two classes. The test case is bundled into the billtest.jar module. The initial class diagram is seen at right. Note the bi-directional relationship between the two classes.

As we progress, we’ll add more classes and abstractions to the system to help increase the modularity. Additionally, we’re going to use JarAnalyzer to illustrate the relationships between the modules and also help us assess the quality of our design. The module structure is below, as generated by JarAnalyzer. You can see how to use JarAnalyzer in the build by reviewing the build file. Again, our goal is to break the cyclic dependency between cust.jar and bill.jar, and we’re going to look at three different ways to do this before moving on to examine different ways to massage acyclic module relationships.

Initial Module Structure with Cyclic Dependencies

Escalation

The first technique we’re going to apply is called Escalation. With Escalation, we break the cyclic dependencies by escalating the cause of the dependency to a higher level entity. Before we do that, we need to more fully understand why a cyclic dependency exists in this example. This reason follows:

A Customer has a list of Bill instances. When the pay method on Bill is invoked, the Bill needs to determine if a discount should be applied. The discount is a product of the Customer the bill belongs to, not necessarily the Bill. Therefore, the Bill class calls a method on Customer to determine the appropriate discount amount. Think of it this way…The Customer represents a payee and we negotiate a discount with each payee. The calculation of this discounted amount is encapsulated within the Customer.

To break this dependency, we want to escalate the cause of the dependency up to a higher level class - the CustomerMediator. The mediator now encapsulates calculation of the discount and passes that to the bill class. The best way to see this change is to look at the modified PaymentTest class. Now, I’ve modified the build script and have bundled the mediator into it’s own module, as shown below. If you dig a bit more deeply into the class structure, you’ll wonder why I didn’t just pass the discount amount from Customer into Bill. Don’t worry about that. This example is slightly contrived because escalation isn’t the best way to solve this type of problem. The key takeaway is that we’ve escalated the dependency up to the mediator.jar bundle, breaking the cyclic dependency.

Escalating the Cause of the Cyclic Dependency

Demotion

A slightly better way to solve this particular type of cyclic dependency (where we have a true composite relationship between Customer and Bill) is to use demotion. With demotion, we push the cause of the dependency to a lower level module. Exactly the opposite of escalation. We do this by introducing a DiscountCalculator class that will be passed into the Bill class. Our modified PaymentTest class will create the calculator and pass it in for us. The Customer class will serve as the factory for the DiscountCalculator, since it’s the Customer that knows the discount that must be applied. The new class structure can be seen at right.

Now we’ll modify our build script to create an additional calc.jar bundle which will contain the DiscountCalculator class. Our resulting module structure is shown below.

Demoting the Cause of the Cyclic Dependency

Already you can see how this is a more natural solution than escalation for this particular type of cyclic dependency problem. What’s the key difference you might ask? With escalation, notice how I would be able to deploy the cust.jar and bill.jar modules independently. While demotion is a more natural solution in this situation, it also means that to deploy bill.jar or cust.jar, I must also deploy calc.jar. The right solution is always going to be contextual and the ideal solution is likely to shift throughout the development lifecycle.

Callback

Using a Callback is similar to the Observer pattern. With this approach, we’ll refactor our DiscountCalculator class to an interface, and then modify the Customer class to implement this interface. This new class structure can be seen at right.

As it happens in this specific situation, using a Callback represents a combination of demotion and our initial solution. We’ll go back to passing the Customer into the Bill, but will pass it in as a DiscountCalculator type. Whereas in the Demotion example we bundled the DiscountCalculator in a separate module, we’ll now just include it in our bill.jar module. Note that putting the DiscountCalculator in the cust.jar module would introduce the cyclic dependency we’re trying to get rid of. The new module structure, which resembles our original version minus the cyclic dependency, is shown below.

Using a Callback to Eliminate the Cyclic Dependency

Inverting Relationships

Now we’re going to play around a bit with the module relationships. While Callback seems like the most logical solution, what if we wanted to use the cust.jar module without the bill.jar module? Callback, as it’s implemented, doesn’t allow us to do this. But with a bit of trickery, I can actually invert the relationship between the cust.jar and bill.jar modules.

I start by refactoring the Bill class to an interface. Then, to avoid split packages (where classes in the same package are bundled into separate modules), I move the Bill class into the same package as the Customer class. The new class diagram is shown at right, and the inverted module structure is shown below.

Inverted Module Structure

Eliminating Relationships

Inverting the relationships allows us to deploy the cust.jar module independent of the bill.jar module. Again, it’s all about need. But I’d like to explore another option based on another important need - the ability to test modules independently. Before inverting the relationships, I am able to test the bill.jar module independently. After inverting the relationships, I can test the cust.jar module independently. But what if I want to test (or deploy) both modules independently? To do this, I need to completely eliminate the relationship altogether.

As it turns out, because I’ve got a pretty flexible class structure after I inverted the relationships (lot’s of abstract coupling), I can do this by simply bundling the two interfaces - Bill and DiscountCalculator - into a separate module. No other coding changes required. I start by moving them to a new package called base. Then, I modify my build script to bundle these two interfaces into a separate base.jar module, and we have successfully eliminated the relationship between the bill.jar and cust.jar modules, as shown below.

Eliminating Relationships Between Modules

Wrapping Up

We’ve come a long ways from our original version of the system. Two modules with a cyclic relationship to a module structure where the original modules don’t have any relationship to each other. This means these modules can be tested and deployed independently. If you follow this blog, you know I’ve written plenty about the tradeoffs between flexibility and complexity, use and reuse, and many other architectural and design challenges. I hope this little exercise has helped drive some of those concepts home.

As a final note, to explore some of these design decisions a bit more deeply, and to examine the tradeoffs a bit more objectively, I encourage you to run the builds for each of the projects and examine the dependencies.html file in the stats directory. To do this, you’ll need to make sure JarAnalyzer is executed, which requires GraphViz. As you’ll see when comparing the initial version to the final version, we made considerable progress in improving the quality of the design.

@kirkk.com blog: A New Year’s Declaration

Tue, 05 Jan 2010 19:15:30 +0000

2010 is going to be a big year for modularity on the Java platform. So based on recent industry trends, and as momentum continues to build around modularity on the Java platform:

I hereby officially declare 2010 as the year of modularity on the Java platform.

So that’s a relatively silly statement, and I really don’t have any authority to make such sweeping declarations. But the statement is backed by some pretty serious trends. Here are a few nuggets to chew on that has led me to believe modularity is going to be bigger in 2010 than ever before!

Between 2007 and 2009, we saw amazing momentum build around OSGi. GlassFish, Jetty, Spring DM, SpringSource dm Server, Paremus Service Fabric, Nimble, PAX, Sigil, Aries, WebSphere, JBoss, Maven, and of course Eclipse are just a few products that leverage OSGi. The list goes on. The number of 3rd party frameworks that have been “osgi-ified” has also grown substantially. And whether you love it or hate it, let’s not forget about Jigsaw and JSR-294.

But there has been a nagging problem…an elephant in the room you might say…that’s hindered widespread adoption. Without gluing together a platform, assembling it yourself, or adopting an entirely new platform, aside from the vendors who were already exposing the virtues of OSGi in their products, the majority of enterprise developers were unable to leverage any of it to build modular enterprise applications! But in 2010, that’s all going to change.

Modularity is coming to the enterprise and you’re going to be able to use it to build modular applications. IBM has already made this announcement. I suspect others will follow suit. For those using SpringSource dm Server or Infiniflow, you’re probably already doing it. With the major platform vendors showing their support for a technology that’s already embedded in products such as dm Server and Infiniflow, I suspect these platforms will see an increase in market penetration through 2010 and beyond. And I suspect you can expect some big announcements surrounding these products soon. Finally, let’s not forget, if it can avoid another delay, JDK 7 (with support for modularity) will be released in September.

Whereas prior to 2010, enterprise developers could only dream about building modular applications (or work very hard to do so), modularity is going to take center stage. And since modularity lies at the heart of the platform, it doesn’t matter what language you use. Neil showed us how to do it with Scala way back in 2007. And Groovy does OSGi, as does Clojure. Yep, so does Scala. So no matter which language you use, you’ll be able to take advantage of modularity on the Java platform.

In 2010 modularity on the Java platform is going to be a game changer…a disruptor…that reaches from the developer to the data center. As 2010 progresses, more and more developers will be touched by modularity. It’s not something we should ignore. Really, it’s not something we can ignore anymore.

@kirkk.com blog: Back to the Future

Wed, 30 Dec 2009 17:30:08 +0000

Thanks

This is my last post of the year, and I want to take a moment to thank everyone that spends their precious time reading my long-winded entries. In 2009, I saw a four fold increase in blog traffic over 2008, and December 2009 saw a five fold increase in traffic over January 2009. Out of a total of 74 posts this year, more than half were related to modularity. I’d like to think the increase in traffic is related to the topics I’ve been writing about. Modularity is something that folks are interested in. But we still have a ways to go.

Future of Modularity

Interestingly, I was speaking to a conference organizer yesterday, and I sensed his mild surprise when I began to talk about modularity. He noted that modularity is certainly not a new concept. I agreed. It’s an idea that has been around since at least the early 1970’s when David Parnas published his essay. But things are different now. Until recently, modularity was not a first class concept on the Java platform. If we wanted to develop software with a modular architecture, we were required to do so without much framework or platform support. In fact, we hadn’t even identified any standard definition of module on the platform.

That’s changing. As the application platform vendors continue to bake OSGi into their products, it’s clear that a module is a JAR file and there’s a framework to back it up. In 2010 modularity on the Java platform will gain considerable visibility in our industry. The stackless stack is coming to fruition, and it’s a game changer…a disruptor…that reaches from the developer to the data center. It’s not something we should ignore. Really, it’s not something we can ignore anymore. So, as we close out one decade and usher in another, modularity will celebrate at least its 40th birthday. Maybe we’re finally starting to get it.

All Posts Page

A final note before closing out 2009. For those interested, I’ve created a custom Wordpress page that shows a summary of all posts published on this blog. Enjoy!

@kirkk.com blog: OSGi DevCon Keynote

Mon, 28 Dec 2009 15:32:03 +0000

I’m excited to be given the opportunity to present the keynote at OSGi DevCon London on February 23, 2010. The event is hosted in conjunction with JAX London. The sessions for the conference look great, with a great lineup of speakers and 15 sessions, including two in-depth tutorials by Neil Bartlett and Peter Kriens, devoted exclusively to OSGi.

Here’s the abstract for my session titled OSGi in the Enterprise: Agility, Modularity, and Architecture’s Paradox.

Attempts to architect more flexible software often results in the opposite - brittle software fraught with complexity. Something is missing. Complexity is the beast we must tame, and modularity is part of the answer. In this keynote, we’ll examine the challenges of traditional approaches to software architecture, probe the paradox of architecture, and explore the inextricable link between structural and temporal architectural decisions. From the highest level applications and services to the code that exists in the bowels of the system, we expose the advantages of a modular architecture. Come learn new ways that large software systems can be organized to increase flexibility, reusability, adaptability, maintainability, testability, and extensibility. Come discover the importance of OSGi in the Enterprise.

Yes, come discover the important of OSGi in the Enterprise!

@kirkk.com blog: That Rotting Design

Mon, 21 Dec 2009 16:26:31 +0000

Note: This is a re-post, with slight modifications, from an entry in October 2007. And now, two years later, we’re just about there! For the abridged version of this post, focus on the text in bold. You can see the original version here.

Design Rot

We’ve all experienced that sinking feeling when maintaining a piece of crappy software. Has my change broken the system in some unintended way? What is the ramification of my change on other parts of the system? If you’re lucky, and the system has a robust suite of unit tests, they can offer some support in proving your work. In practice, however, few systems have thorough automated test coverage. Mostly we’re alone, left to verify our changes as best as possible. We might privately criticize the original developers for creating such garbage. It certainly lends a plausble excuse in explaining why the maintenance effort is so costly or time-consuming. Or it might serve as the basis upon which we recommend a re-write. But mostly, we should wonder how it happened.

For sure, most software doesn’t start out this way. Most software starts out clean, with a clear design strategy. But as the system grows over time, strange things begin to happen. Business rules change. Deadline pressures mount. Test coverage slips. Refactoring is a forgotten luxury. And the inherent flaws present in every initial design begin to surface. Reality has proven that few enterprise development teams have the time or resources to fix a broken design. More often, we are left to work within the constraints of the original design. As change continues, our compromises exacerbate the problem. The consequence of rotting design is seen throughout the enterprise on a daily basis. Most apparent is the affect on software maintenance. But rotting design leads to buggy software and performance degradation, as well. Over time, at least a portion of every enterprise software system experiences the problem of rotting design. A quote from Brook’s sums it well:

All repairs tend to destroy the structure, to increase the entropy and disorder of the system. Less and less effort is spent on fixing the original design flaws; more and more is spent on fixing flaws introduced by earlier fixes. As time passes, the system becomes less and less well-ordered. Sooner or later the fixing ceases to gain any ground. Each forward step is matched by a backward one. Although in principle usable forever, the system has worn out as a base for progress.

The most obvious question is, “How do we prevent rotting design?” Unfortunately, rotting design is not preventable, only reducable. Of the past ten years, the design patterns movement has provided insight to the qualities of good design. Dissecting design patterns reveals many important design principles that contribute to more resilient software design. Favor object composition over class inheritance, and program to an interface, not an implementation, are two examples. Of the 23 patterns in the GOF book, all adhere to these fundamental statements. Alone however, design patterns that emphasize class structure are not enough to help reduce rotting design.

Reducing Rot

Most patterns emphasize class design, and present techniques that can be used in specific contexts to minimize dependencies between classes. Teasing apart the underlying goal of most patterns shows us that each aim to manage the dependencies between classes through abstract coupling. Conceptually, classes with the fewest dependencies are highly reusable, extensible, and testable. The greatest influence in reducing design rot is minimizing unnecessary dependencies. Yet enterprise development involves creating many more entities beyond only classes. Teams must define the package structure in which those classes live, and the module structure in which they are deployed. Increasing the survivability of your design involves managing dependencies between all software entities - classes, packages, and modules.

But if minimal dependencies were the only traits of great design, developers would lean towards creating very heavy, self-contained software entities with a rich API. While these entities might have minimal dependencies, extreme attempts to minimize dependencies results in excessive redundancy across entities with each providing its own built-in implementation of common behavior. Ironically, avoiding redundant implementations, thereby maximizing reuse, requires that we delegate to external entities, increasing dependencies. Attempts to maximize reuse results in excessive dependencies and attempts to minimize dependencies results in excessive redundancy. Neither is ideal, and a gentle balance must be sought when defining the behavior, or granularity, of all software entities - classes, packages, and modules. For more on the use/reuse paradox, see Reuse: Is the Dream Dead?

Software design is in a constant quandary. Any single element key to crafting great designs, if taken to its individual extreme, results in directly the opposite - a brittle design. The essential complexity surrounding design is different for every software development effort. The ideal design for a software system is always the product of it’s current set of behavioral specifications. As behavior changes, so too must the granularity of the software entities and the dependencies between them. The most successful designs are not characterized by their initial brilliance, but instead through their ability to withstand the test, and evolve over the course, of time. As the complexity of software design is an essential complexity surrounding software development, our hopes lie with technologies and principles that help increase the ability of your design to survive. Such is the reason why agile architecture is so important.

A Promising Future

I’m hopeful that all software developers have experienced the pleasure of a design that, through the course of time, has withstood the test of time. Unfortunately, many enterprise development teams have too few of these experiences. Likewise, few enterprise development teams devote adequate effort to package and module design. It’s unreasonable to believe that even the most flexible class structure can survive should the higher level software entities containing those classes not exhibit similarily flexible qualities. The problems are rampant. Increased dependencies between packages and modules inhibit reusability, hinder maintenance, prevent extensibility, restrict testability, and limit a developer’s ability to understand the ramification of change.

Services offer some promise to remedy our failures with object-oriented development. Yet, while services may offer tangible business value, within each awaits a rotting design. There exists a world between class design and web services that deserves more exploration, and as an industry, we are beginning to notice. OSGi is a proven module system for the Java platform, while Jigsaw aims to modularize the JDK. JSR-294 aims to improve modularity on the Java platform. While some friction might exist between the constituencies involved, it’s only because they too recognize that something has been missing, and are passionate about fixing the problem. Of course, it doesn’t stop there. A plethora of application servers and tools are also including support for modularity using OSGi, which has grown into the defacto standard module system on the Java platform.

All aim to help manage the complexity, from design through deployment, of enterprise software development. With each, new practices, heuristics, and patterns will emerge that increase the ability of a design to grow and adapt.

@kirkk.com blog: Top 5 Essays You Should Read

Thu, 17 Dec 2009 16:14:31 +0000

Certainly, there are many landmark books in software development that have shaped our industry. Design Patterns: Elements of Reusable Object-Oriented Software (the GOF book) is one really good example. Unfortunately, a good share of people don’t have the opportunity to read some of these great works because, well…it can get expensive pretty quickly stocking a bookshelf. But there exists a treasure trove of published content available online that is equally impactful. Here, in no particular order, are 5 essays that have helped shaped our industry, for better or worse.

Cathedral and the Bazaar by Eric Raymond - Discusses the evolution of Linux and provides amazing insight to lessons learned.
Code as Design by Jack Reeves - Presents the notion that programming is fundamentally a design activity and that the only final and true representation of “the design” is the source code itself.
Managing the Development of Large Software Systems (pdf) by Winston Royce - Paper widely regarded as that which gave birth to the waterfall development lifecycle.
No Silver Bullet by Frederick Brooks - We’re still looking, but as this paper points out, there is no silver bullet. The essential complexity Brook’s speaks of is largely why we continue to struggle with the same problems today that we did a decade ago.
On the Criteria to be Used in Decomposing Systems into Modules by David Parnas - Discusses the important design decisions that impact how we modularize our software systems. Important because modularity is coming to the Java platform, and we need to know how to use it effectively.

I’ll give an honorable mention to Design Principles and Design Patterns by Bob Martin, which discusses key principles of object-oriented design. Many of the patterns in the GOF book adhere to these principles.

These essays are sure to provide a positive and lasting influence. But I’m sure there are more. What am I missing? What do you consider the most impactful software development essays? What would you add to this list?

@kirkk.com blog: Architecture All the Way Down

Tue, 15 Dec 2009 00:00:22 +0000

In Turtles and Architecture, I talked about how important it is that we “architect all the way down”. It helps increase transparency and understanding between developers and architects by emphasizing a lot of the middle ground that noone ever seems to focus on. It’s just another reason why modularity is so important. I used the diagram at right to illustrate the point (click to enlarge). Look at the huge gap that exists if we don’t focus on the stuff highlighted by the gray bar in the middle.

One reason I like this visual is that it illustrates the social aspect of software architecture. Yet, there are other significant advantages to architecture all the way down that we haven’t explored yet. Another is structural flexibility.

Structural Flexibility - Different Entities, Different Purpose

Another benefit of module design in filling that middle ground is that modules can offer different benefits than classes and services. The diagram at right (click to enlarge) illustrates some of the capabilities of different types of entities.

For example, classes are pretty easily reused within an application, but because classes aren’t a unit of deployment, it’s difficult to use them across applications. Of course, intra-application reuse is a sweet spot of services, since they’re method of invocation is through some distributed protocol (SOAP, HTTP, even RMI/IIOP). Yet because services are invoked remotely, we typically like to make them coarser-grained because of the performance implications of distributed requests. So this begs the question - If a service is too coarse-grained to reuse (ie. it does more than what we want), how do I reuse some desirable behavior across applications? Well, without modules, our only other choice is the class. Since a class can’t be reused intra-process, we do one of either two things. Expose it as a service or copy the class (ie. the code). Given the circumstances, neither of these may be ideal. Another option is desirable.

Modules represent that other option. They are a finer level unit of granularity than services, and are a unit of deployment. Since each of these different types of entities are units of composition, we have tremendous flexibility in how we assemble applications. Possibly more important though is our increased ability to accommodate the architectural shifts that naturally occur as requirements evolve. Let’s look at a more concrete example.

A Bit More Concretely

Let’s say I have a business function called Pay Bill for which I develop a web service that can be invoked by many different consumers. That service happens to be relatively coarse-grained, and performs all the steps involved in paying the bill. These happen to include the following:

audit bill - apply a discount to the bill based on payee
check for duplicate - ensure the bill hasn’t already been paid
remit payment - cut the check
reconcile payment - reconcile with accounts payable financials

This seems reasonable. We have a nice little service that we can reuse any time we want to pay a bill. Unfortunately, the real world often gets in the way of our idealistic solutions. In fact, there are two problems that will eventually surface, and modularity benefits both scenarios. Let’s start by looking at the first scenario.

What should I do when a different scenario arises that demands I follow a slightly modified Pay Bill function?

As part of the remit step, I have a new payee that demands electronic payment. This is pretty easy actually. I simply modify the service to support electronic payments and then configure the service to context for that specific payee. So how does modularity help here?

If the service is composed of modules, it’s going to be much easier for me to understand the structure of the service, assess the impact of what’s it’s going to cost to change the service, and then introduce a new module (or modify the existing module) to provide the new capability. Without modules, I’m simply wading through the code trying to figure out all of these things. Now, the 2nd scenario.

What should I do when I want to reuse just one step of the Pay Bill function?

Let’s say another new requirement emerges. Whereas traditionally bills were entered by data entry personnel, we now have to support electronic delivery of bills. We also know that bills delivered electronically are often duplicates. It’s just one of those business things, you know? If we don’t pay the bill on the day it’s received, the billing party sends us the bill again, asking for payment. So we need to check for duplicates before we save the bill to the database and prepare it for processing. What do we do?

We could take the same approach as before and modify the Pay Bill service so that the duplicate check could be invoked separately from the higher level pay bill function. But that’s a bastardized design solution. Why? We are exposing behavior of the Pay Bill service that shouldn’t be exposed. The API is coarse-grained in some areas and fine-grained in others.

Maybe exposing the finer-grained capabilities of the Pay Bill function isn’t a severe compromise, but it is a compromise nonetheless. And we are forced to compromise because of architectural inflexibility. We don’t have architecture all the way down, and are therefore left with limited choice as to how we support the new requirement. We can either modify the service to reuse what we know is already encapsulated within the service, or copy the code to create something new. But those are the two options we have, and neither may be ideal.

As we continue to hack the system in this manner, with a multitude of other similar changes, the design will rot. Eventually, our Pay Bill service is transformed into a utility that performs all general-purpose bill functions. It’s API is a mix of coarse-grained and fine-grained operations. It’s become the monolith that we’re trying to avoid. While the Pay Bill service is still pretty reusable (it does everything now), it isn’t that usable. That tension between reuse and use surfaces again.

Our decision to modify the Pay Bill service to expose the duplicate check was driven by one thing - ease. It was our easiest option. Really, it was our only option. But it isn’t the best option.

If we architect all the way down, we have another option. A Pay Bill service composed of modules that audit the bill, check for duplicates, remit payment, and reconcile the payment means we can choose the desirable solution over the easiest short term choice. We can avoid the long term compromises that degrade architectural integrity.

Shown at right (click to enlarge), we see the service composed of modules. If we have a check for duplicates module, we can simply reuse that module in the new service or application that’s going to process electronic bills. Or we might expose the capabilities of the check for duplicates module as a separate service. We have multiple reuse entry points, as well as different levels of granularity through which our software entities can be managed, deployed, built, and much more.

To Conclude

My point here isn’t to debate the original design nor the decisions made to reuse the check for duplicates functionality in another service or application. Certainly, debating architectural and design decisions is a favorite past-time of developers. There are a variety of different ways that we can support new requirements as they emerge. Some are better than others and all are contextual.

The gist of my point is that architecture all the way down gives us options, and these options help maintain architectural integrity. They increase our options when making decisions and allow our system to accommodate unforeseen architectural shifts. I can modify an existing application or service to give me what I need. Or I can reuse an existing module that’s already available for me. Or I can compose a new service from an existing set of modules. Or I can break apart existing modules to create new modules that result in new services. And I can do a lot of this refactoring without significant impact on the existing code. This was an important point illustrated in the series of posts on Applied Modularity.

In other words, architecture all the way down helps increase architectural agility, and modularity is a key ingredient.

@kirkk.com blog: The Secret Sauce

Tue, 08 Dec 2009 00:45:05 +0000

All too often, software process improvement initiatives fail. In a recent post discussing SEMAT, Ralph Johnson provided some words of wisdom that serve as a wonderful guide to any team about to embark on that much vaunted software process improvement initiative.

The state of the practice in software development is pretty dismal. Some groups do a great job, but most do not. As I tell the students in my software engineering course, if you manage requirements, make sure the developers talk to each other, release working code regularly, have some sort of a systematic testing process, use build and version control tools, and periodically stop and see how you are doing and how you can improve, you will be better than 90% of the groups out there. Of course, I could be exaggerating. Maybe it is only better than 75%.

I suppose that pretty much sums it up! Amazing how difficult we tend to make things though, heh?

@kirkk.com blog: The Enterprise is Getting Its OSGi

Fri, 04 Dec 2009 14:55:00 +0000

There have been a few folks who believe that OSGi isn’t something that enterprise developers want. I’ve heartily disagreed with this, and have stated that the enterprise wants it’s OSGi. In The Two Faces of Modularity and OSGi, I stated:

While there is some dispute over whether vendors will expose the virtues of OSGi to the enterprise, eventually they will (SpringSource dm Server currently does). They will because the enterprise stands to gain considerably from a more modular architecture. There is benefit in a world void of classpath hell and monolithic applications. There is benefit in modularity, and eventually the enterprise will ask for OSGi. In the end, the enterprise will get what they’re asking for…

Well, it looks like that day is finally arriving. IBM recently announced the IBM Websphere Application Server V7 OSGi Application Open Alpha Program. From the program’s homepage, I quote the following:

The OSGi Applications Open Alpha brings the modularity, dynamism, and versioning of the OSGi service platform to enterprise application developers. It delivers a simple to use, lightweight programming model for web applications that combines the standard Blueprint component model with familiar Java enterprise technologies. TheOSGi Applications Open Alpha enables enterprise applications to be deployed to the WebSphere Application Server as collections of OSGi bundles, leveraging WebSphere platform enterprise qualities of service to provide the most complete and robust enterprise server for modular web applications. In addition to being easy to use, theWebSphere OSGi applications implementation addresses many of the challenges of developing and maintaining extensible Web applications through OSGi technology benefits of modularity, dynamism, and versioning.

Additionally, a post to the IBM developerWorks forum discusses some of the capabilities, which includes:

a built-in bundle repository which can host common and versioned bundles shared between multiple applications so that each application does not deploy its own copy of each common library.
a standardized (by the OSGI Alliance) and IBM-supported evolution of the Spring framework called the Blueprint component model. This provides the declarative assembly and simplified unit test of the Spring framework but in a standardized form that is provided as part of the WAS runtime rather than being a 3rd party library deployed as part of the application
familiar Java EE programming model with the option of deploying web applications as a set of versioned OSGi bundles with dynamic lifecycle.
full integration with WAS Admin and WAS platform enterprise qualities of service

This is great for enterprise developers who are tied to the Websphere Applicatin Server (WAS) platform. But don’t confuse WAS support of OSGi with WAS being a next generation application platform. Other vendors have done a great job providing platforms built atop OSGi, including SpringSource and Paremus.

And don’t confuse support of OSGi as a promise that you’ll immediately realize the benefits of modularity. We’ve made that mistake before with other technologies. There is still a lot to learn about how to design more modular software applications. While WAS support for OSGi brings many of the runtime and management benefits of modularity to the enterprise, it’s still our responsibility to craft systems with a modular architecture.

I’m still unsure of the actual release date for WAS V7, and I don’t have information surrounding all the prickly details of OSGi in WAS 7 and the Open Alpha program. But I intend to find out, and have already reached out to IBM to gather more details. There is still more to learn, so stay tuned. Maybe…just maybe…2010 will be the year that OSGi (and modularity) hits the enterprise.

@kirkk.com blog: Applied Modularity - Retrospectives

Thu, 03 Dec 2009 15:26:03 +0000

We’ve completed our four part series on Applied Modularity, and I wanted to put a final wrap on the posts by highlighting a few things that may not be obvious. First, a brief review on the series.

In Part 1, we introduced the sample application and applied PhysicalLayers pattern to separate our logical layers out into physical layers.
In Part 2, we applied the AbstractModules (second refactoring) and AcyclicRelationships (third refactoring) patterns.
In Part 3, we applied the SeparateAbstractions (fourth refactoring) pattern.
In Part 4, we applied the CollocateException (fifth refactoring), IndependentDeployment (sixth refactoring), and ImplementationFactory (sevent refactoring) patterns.

Through this series of refactorings, we made considerable progress in modularizing the application using a few of the modularity patterns. The diagram at right (click to enlarge) illustrates the progress we made. We started with everything bundled into a single WAR file and wound up with a highly modularized system that satisfied the evolutionary requirements. Aside from the many advantages we spoke about in each post, I want to take a moment to explore a few other thoughts.

A Note On Module Testing

If you’ve explored (and built) the system by getting the code from the Google code repository, you’ll notice that there are a corresponding set of test modules for each module that we’ve created. These can be found in the bin directory (shown at right). Like we do with unit testing, I’ve tried to create a test component for each module in the system. Unfortunately, there’s a flaw in the billtest.jar module.

Similar to unit testing, where we create mocks and stubs to avoid undesirable dependencies, a test module shouldn’t pull in other modules that contain implementation classes. Instead, we should create mocks or stubs to avoid this situation. In other words, a test module should only be dependent on the same set of modules as the module it’s testing. Unfortunately, the billtest.jar module breaks this rule by leveraging the the AuditFacade implementations. That means the billtest.jar module is also dependent on the audit1.jar and audit2.jar modules, but the bill.jar module is not. So billtest.jar is really a module integration test, not a module unit test. It could easily be fixed by creating a mock AuditFacade implementation that lived in the billtest.jar module.

This begs another question….

How do we keep track of module relationships so that we recognize when something bad like this happens?

Even for small systems, without a module system like OSGi, it can be incredibly challenging.

A Note On Managing Modules

Modularizing a system on a platform that doesn’t support modularity is challenging. Hell, modularizing a system on a platform that does support modularity is challenging! One of the greatest challenges is in managing module dependencies. Tracking the dependencies between modules is really quite difficult.

This is where module systems like OSGi really help by enforcing the declared module dependencies. In plain ole Java today, there is no notion of module so there is nothing to help enforce modularity. And the first unwanted dependency that creeps into our system compromises architectural integrity. This is where JarAnalyzer can be helpful. By incorporating JarAnalyzer into my build script, I’m able to more easily manage the dependencies between modules.

JarAnalyzer has two output formats. The first is a GraphViz compliant dot file that can be easily converted to an image showing module relationships. The image at right (click to enlarge), which includes the test modules, clearly illustrates the problem with the billtest.jar module discussed above.

As can be seen, the bill.jar module has only a single outgoing dependency on the auditspec.jar module. So the module that tests the bill.jar module should not be dependent on any other modules, either. However, if you look at the billtest.jar module, you’ll see that it depends upon the audit1.jar and audit2.jar modules. So instead of using a mock or stub to test the bill.jar module, I got lazy and used the various AuditFacade implementations. Look at a few of the other modules, and you’ll discover that none include additional dependencies beyond the dependencies already present within the modules they test.

The second output format for JarAnalyzer is an html file that provides some key design quality metrics, as well as listing the dependencies among modules. Essentially, it’s a textual view of the same information provided by the visual diagram. I’ve included the Summary header of the JarAnalyzer report for the system below (click to enlarge). You can also browse the complete JarAnalyzer HTML report for the final version of the system. There is also a version that omits the test modules.

Look at the auditspec.jar module. Note that it has 8 incoming dependencies (afferent coupling) and 0 outgoing dependencies (efferent coupling). It’s abstractness is 0.67 and Instability is 0.00. This is a pretty good sign. Why? It’s instability is very low, implying it’s highly resistant to change. It possesses this resistance to change because of the large number of incoming dependencies. Any change to this module may have serious implications (ie. the ripple effect of change). But because it’s quite abstract, it’s less likely to change than a module with a lot implementation classes. The Distance for the module is 0.33 (ideal is 0.00), so we’re not far from where we ideally want to be.

In case you’re wondering about all these metrics I’m rambling about, you might want to take a look at the Martin Metrics. In general, without a utility like JarAnalyzer (or a module framework like OSGi), it would have been incredibly difficult to manage the modules composing this system.

A Note on Module Reuse

The reuse/release equivalency principles states that the unit of reuse is the unit of release. Modules are a unit of release, and therefore are a unit of reuse. Naturally, the devil is in the details, and we’re going to discuss these details here.

In Reuse: Is the Dream Dead, I spoke of the tension between reuse and use. That tension is evidently at play here. Earlier versions of the system had coarser-grained modules that were easier to use but more difficult to reuse. As we progressed, we broke these coarser-grained modules out into finer-grained modules, increasing their reusability but decreasing their ease of use. A perfect example of this is the bill.jar module. In the final version, it was quite reusable, since it was only dependent on the auditspec.jar module. However, this came at the price of useability.

To elaborate a bit more. In Part 4, the sixth refactoring, we decoupled the bill.jar and financial.jar modules so the two could be deployed independently (ie. increase reuse). But the runtime structure still has some dependencies. In order to reuse bill.jar, we need a BillPayer type. While an alternative BillPayer implementation could be created, the existing implementation is the BillPayAdapter in the mediator.jar module, which also has a relationship to the financial.jar module. This means that to use the bill.jar module without the mediator.jar and financial.jar modules would require a new consuming module to implement the BillPayer interface.

So what do we do if we want to break this runtime coupling? We should move the pay method on the Bill up to the BillPayAdapter class, and get rid of the BillPayer interface. Now the Bill class has no dependency on the BillPayer interface, but it also can’t make payments. Every action has an equal an opposite reaction, heh?

A Note on The Build

The build was a key element in helping enforce modularity (note: JarAnalyzer helped me manage module relationshps; the build enforced module relationships). Even a framework such as OSGi is only going to manage module relationships at runtime. I talk a bit more about this concept in The Two Faces of Modularity & OSGi, and it’s why we need really good tools that help us design more modular software. It’s our responsibility to craft the modules, and the build is one way to help put in place a system of checks and balances that help enforce modularity before discovering at runtime that one module has a relationship to another. In Part 2, as part of the third refactoring, we refactored our build script to a levelized build. Here’s the before and after build script.

This means that as we build each module, we include only the required modules in the build classpath. This is more easily explained by examining the build script for the final version, where you can clearly see what I’m talking about. Look at line 40. When we build the auditspec.jar module, we include nothing else in the build classpath because the auditspec.jar module doesn’t require anything. Now look at line 60, where we build the audit1.jar module. The auditspec.jar module built in the previous step is included in the classpath. This pattern recurs throughout the remainder of the script. Introducing a module dependency that violates the dependency structure enforced by the build results in a failed build.

A Note on Object Orientation

The way we managed, massaged, and modified module relationships was through OO techniques. By introducing interfaces and abstraction and allocating them to their respective modules, we were able to significantly change the module structure of the system. While we used OO to do this, OO is not a prerequisite. We could just as easily have used other techniques, such as aspects (AOP).

The Final Wrap

If you’ve made it this far through the tutorial, you’ve done well. Overall, this was a pretty lengthy and involved tutorial. In fact, I only touched briefly on all that I really had to say. Yeah, you’ve seen the abridged version here! I think I could pretty easily fill a book with the rest. Hmmm…

I use this same system and examples in quite a few of my talks on architecture and modularity. If you have questions or suggestions, feel free to drop me a line via the comments or send me an e-mail (hint: look on the About Page). Or you can track me down at a conference, as I’m always happy to discuss topics related to modularity, architecture, and agility.

I have one final deliverable. As promised way back in Part 1, I intend to show an OSGi-ified version of the system. That will follow shortly.

@kirkk.com blog: Applied Modularity - Part 4

Mon, 30 Nov 2009 19:25:38 +0000

In part 3, we applied the SeparateAbstractions pattern so that we could independently manage the two different AuditFacade interface implementations. The result was the module relationships shown at right (click to enlarge). The code can for this version of the system can be found in the SeparateAbstractions project. Also, be sure to check out Part 1, where we introduced our initial version of the system, and Part 2 where we applied a couple of the modularity patterns. Now, we’re going to focus on the financial.jar module and try to decouple bill.jar from financial.jar so the two are completely independent of each other.

I think you’ll really like this one…But first, some additional housekeeping.

Fifth Refactoring

Before we decouple the bill.jar module from the financial.jar module, we’re going to talk about exceptions. In general, we need to answer the question.

Where do exceptions belong?

In the CollocateExceptions project, we introduce an AuditException that is thrown if an error is encountered. The CollocateExceptions pattern says that exceptions should be close to the classes (or interfaces) that throw them. Since the AuditFacade interface throws the exception (shown below), we should put the AuditException in the same module as the AuditFacade interface, which is the auditspec.jar module. Note that if we put the exception anywhere else, we’d create an unwanted dependency between modules.

Sixth Refactoring

Ok, the previous refactoring was pretty simple, and it’s time to move on. Here’s the need, and I think you’ll find the solution quite interesting.

A new requirement has emerged, and we need to use the bill.jar module in another system. In this other system, we don’t need the financial.jar module to make the payment. We just want to use the bill.jar functionality. So, what do we do?

First, recall our initial version of the class structure of the system. In the third refactoring in Part 2, we introduced a Payable interface, and then created the financial.jar module. After we finished Part 3, we were left with the module structure shown at right (click to enlarge). Again, our goal here is to eliminate the dependency between the bill.jar and financial.jar modules so that we can deploy the bill.jar module without the financial.jar module. We’re going to apply a little trick called escalation. Essentially, we’re going to escalate the dependency to a higher level module. How? First, let’s take a look at the new class structure that’s going to allow us to remove the module dependency.

As shown at right (click to enlarge), we see the new class structure that’s going to allow us to decouple the bill.jar and financial.jar modules. Things are starting to get a bit tricky here, so before we allocate these classes to their respective modules, let’s walk through the steps to illustrate how these classes will callaborate to give us what we need.

We start by passing a BillPayAdapter instance to the Bill as a BillPayer type. The BillPayAdapter is going to manage the relationship between Bill and Payment. When the pay method on Bill is called, it will turn around and invoke the generateDraft method on BillPayAdapter. The BillPayAdapter invokes the generateDraft method on Payment, passing itself in as a Payable. This allows the Payment to callback on the Payable and invoke the appropriate method (either getAmount or getAuditedAmount) on BillPayAdapter. The BillPayAdapter can now query the Bill to obtain the audited amount and pass it back to the payment. Once the Payment has this amount, it can make the payment. Whew! A fairly complex callaboration, but enough to ensure we can decouple the bill.jar module from the financial.jar module. The code for the new BillPayAdapter class, which mediates (hmm..possibly naming it a mediator would have been better, huh?) the relationship between Bill and Payment, is shown below.

Now it’s time to allocate these classes to the appropriate modules. The key decision here is where we place the BillPayAdapter. Because this controls the Bill and Payment callaboration, we can’t put it in either of those two modules. In fact, we’re going to create a new module that contains the BillPayAdapter. The new module, which we’ll call billpay.jar, will depend on both the bill.jar and financial.jar modules, as shown at right (click to enlarge). The code for this version can be found in the IndependentDeployment project.

For the most part, our system is complete now. But we do have one nasty little problem that we haven’t solved yet. We have two different AuditFacade implementations, but the current system hardcodes creation of AuditFacade1 within the A uditAction class. All this flexibility compromised by a single line of code. Unnerving how that happens! We really do have to architect all the way down. Anyway, let’s fix this problem.

Seventh Refactoring

The problem of creating instances is pretty common, so the solution is relative easy. There are a lot of different ways to do this, but we’re going to take the easiest route (and not the most flexibile mind you) by creating an AuditFacadeFactory class, which can be found in the ImplementationFactory project. Now, instead of the AuditAction creating the AuditFacade implementation, it invokes the factory and is returned an instance, as shown below.

Wrapping Up and Getting Ready for the Postmortem

We’ve made considerable progress. Amazing really. From a single monolithic application to a fully modularized architecture with considerable flexibility. But there are a lot of very interesting takeaways that we haven’t talked about yet. A number of positive side affects have resulted that aren’t immediately obvious. In our final post in this series, we’ll take a more in-depth look at the impact of modularity.

@kirkk.com blog: Applied Modularity - Part 3

Thu, 19 Nov 2009 04:07:07 +0000

In Part 1, we introduced the system, and broke each layer out into separate modules. In Part 2, we applied two refactorings using two different modularity patterns - AbstractModules and AcyclicRelationships. But we were still left with a problem, which we’ll solve here.

Fourth Refactoring

After the third refactoring, we were left with the modules shown at right (click to enlarge). But this doesn’t entirely solve our problem of how we deploy a new AuditFacade implementation, especially if we want to avoid redeploying the audit.jar module and everything in it. Additionally, because the two implementations of AuditFacade live in the same module, we can’t rip one out (ie. uninstall) when we add the other. In other words, because they are in the same module, they can only be managed together.

To solve this tricky little challenge, we’re going to apply the SeparateAbstractions pattern, which states that we should separate abstractions from the classes that realize them. If you’ll recall from part 2, we applied this pattern when allocating the Auditable interface to the audit.jar module, but we didn’t apply it to the AuditFacade1 class. So we’ll apply it to the AuditFacade1 implementation, as well as a new AuditFacade2 implementation. First, we have to answer the following question.

Where do we put the AuditFacade interface so that new implementations of the interface can be managed separately from the interface and other implementations?

Our answer can be found by looking at the module structure shown at right (click to enlarge). Note that I’ve also included the financial.jar module in this diagram, which was actually introduced in Part 2 when we introduced the Payable interface. We’ll use this soon, but I digress.

Anyway, we separate the AuditFacade interface out into its own module - auditspec.jar. Each of the implementations (AuditFacade1 and AuditFacade2) are also placed in their own modules. The code for this solution can be found in the SeparateAbstractions project. No real coding changes are necessary, other than we’ve added a new class - the AuditFacade2 implementation. Note that we’ve also separated each implementation out into different packages to avoid splitting packages across modules. After adding this new class, we modified our build script to allocate the classes to the appropriate modules.

A Note on the Benefit of OSGi

Now, we have an overall increase in architectural flexibility. Our AuditFacade implementation classes are allocated to separate modules, allowing us to manage the two independently. From a maintenance perspective, we can work on each of the implementations separately, resting assured that changes to one implementation won’t negatively impact changes to the other. We can also test each independently and reuse each independent of the other.

This change also increases the resiliency of the bill.jar module since it’s no longer tightly coupled to any AuditFacade implementation. We can test the bill.jar module using only the auditspec.jar module by creating a mock implementation of AuditFacade for testing purposes. While the billpay.jar module is still dependent on the financial.jar module, we’re going to solve that problem in the next refactoring. In general, we have completely eliminated the dependencies between the bill.jar module and the AuditFacade implementations in the audit1.jar and audit2.jar modules. Remember, at the beginning of Part 2, these modules containing the bill and audit behavior had a cyclic relationship.

While this example doesn’t leverage OSGi, it is important to point out the benefit that OSGi can provide here. Because we don’t have a runtime module system we still need to deploy these modules within the WAR file. The presence of OSGi, however, brings the same degree of flexibility to the runtime that we have at development time (for more information, see The Two Faces of Modularity & OSGi). With OSGi, I would be able to install and uninstall the audit1.jar and audit2.jar modules without redeploying the entire system. When we’re finished with all of our refactorings, we’ll “osgi-ify” the system, deploy it to Tomcat, and realize this benefit.

Wrapping Up and Getting Ready for Part 4

So we’ve made quite a bit of progress from our initial version of the system that completely lacked modularity. By breaking the system out into modules, we ease the maintenance effort and increase overall system flexibility. In part 4, we’re going to to turn our attention to the financial.jar module that focuses on payment. Recall that in Part 2 (when we applied the AcyclicRelationships pattern to the AcyclicRelationships version of the project) , we separated the Payment class out into a separate module and decoupled Bill and Payment through the Payable interface. Part 4 will examine how we can decouple the bill.jar module from the financial.jar module so bill.jar can be managed, tested, and deployed independent of financial.jar. Stay tuned!

@kirkk.com blog: Applied Modularity - Part 2

Thu, 19 Nov 2009 04:05:31 +0000

See Part 1 for an overview of how we got started. In this post, we’re going to take the modularization of our application a step further. We’re going to apply two refactorings using two different modularity patterns - AbstractModules and AcyclicRelationships. First, we’re going to separate the bill and audit functionality out into separate modules so we can manage (develop, deploy, etc.) them independently. Second, we’re going to remove the cyclic dependency between these two modules.

A Quick Review

Before we get started with this second installment, let’s review what we did in Part 1. Initially, our billing application was all packaged and deployed in a single WAR file. Since we had a conceptually layered application, it was pretty easy to separate the UI layer from the business object and data access layer. Not sure we explicitly pointed it out in Part 1, but that refactoring essentially applied the PhysicalLayers pattern. After doing this, we were left with the system at right (click to enlarge).

Second Refactoring

Recall the initial class diagram, where we had a Bill class with a bi-directional relationship to the AuditFacade class. This design has two fundamental flaws. The Bill is tightly coupled to the concrete AuditFacade class and the relationship is bi-directional. Bad all around! This can be seen in the following code snippet illustrating the Bill’s audit method.

Notice that the audit method actually creates the AuditFacade, calls the audit method, and passes a reference to Bill. Ugly. Let’s clean this up a little bit. While there are obvious technology reasons why we need to clean this up, there is also a motivating business force.

The system needs to go live with the current vendor’s auditing system, but the business has indicated that they aren’t renewing the contract with the vendor and are in ongoing negotiation with another vendor. The contract expires in 6 months, but we deliver the initial version of the system in 3 months.

So, 3 months after deployment, we know we’ll need to swap out auditing systems. What to do?

The first half of this solution can be found in the AbstractComponents project, where we apply the AbstractModules pattern which states that we should depend upon the abstract elements of a module. So we’re going to refactor the AuditFacade to an interface and create a separate AuditFacade1 implementation. This solves the first half of our problem - the tight coupling between the Bill and AuditFacade implementation. The result is the class diagram shown at right (click to enlarge). Take a look at the refactored Bill class, and you’ll see that the AuditFacade interface is now passed into the Bill, allowing us to swap out AuditFacade implementations. Now we need to decide how to modularize the system.

If we take this flexible class structure and continue to deploy in the single bill.jar module, we have the flexibility to swap out AuditFacade implementations at the class level, but we’re still required to package everything up into a single bundle and deploy it together. So what we really must do is separate the audit functionality out into separate modules from the bill functionality. Separating the AuditFacade interface and AuditFacade1 implementation out into separate modules results in the diagram at right (click to enlarge). We can also see this change reflected in the build file that packages up these modules. Here’s where the bi-directional relationship between Bill and the AuditFacade interface (and AuditFacade1 implementation) rears it’s ugly head. We have a cyclic dependency between our bill.jar and audit.jar modules. We need to fix this problem.

Third Refactoring

The second half of this solution can be found in the AcyclicRelationships project. Now, we need to remove the cyclic dependency between the modules, and we’ll apply the AcyclicRelationships pattern which states that module relationships must be acyclic. To do this, we’ll introduce an additional abstraction, called Auditable, that our Bill class implements. Upon applying this little trick, we can see that we have now removed the bi-directional relationship between Bill and the AuditFacade interface, as shown at right (click to enlarge).

Whereas previously the AuditFacade accepted a Bill to the audit method, it now accepts an Auditable type. The new AuditFacade interface can be seen in the code snippet below.

The Bill will pass itself to the AuditFacade interface as an Auditable type. The key element at this point is how we allocate these classes to their respective modules. Here’s a simple rule to abide by when determining how to allocate classes and interfaces to modules.

Interfaces should be closer to the classes that use them, and farther away from the classes that implement them.

We’ll talk more about this in the next post, because we build on this idea to address another problem that has surfaced. But applying this rule now, it’s clear that Auditable should be bundled with the AuditFacade interface, not in the same module as the Bill class.

While we haven’t talked about the new financial.jar module, we applied a similar refactoring to the Payment class by implementing a Payable interface, and breaking it out into a separate module. Allocation of classes to modules is done at build time, so after modifying our build script, the result is the modules shown at right (click to enlarge).

As I’m sure you’ve noticed at this point, we broke our rule above and bundled the AuditFacade1 implementation close to the interface it implements (btw, this is the other problem that has surfaced). If we apply our rule in this situation, we’d bundle the AuditFacade interface and Bill class in the same bundle, which certainly wouldn’t work because it would result in the cyclic dependency between the bill.jar and audit.jar modules that we’ve worked so hard to remove. Alas, this is the focus of our next refactoring, so we’re going to wait to tackle this challenge.

Wrapping Up and Getting Ready for Part 3

This post involved two refactorings. The first was to separate the bill and audit functionality out into separate modules. The second was to remove the cyclic dependency between the bill and audit modules.This offers us decent flexibility. We can introduce a new AuditFacade implementation, knowing that we’d only need to rebuild the audit.jar module. We can test the audit.jar module independent of any other module because it doesn’t have any outgoing dependencies. And if we wanted to, we could deploy the audit functionality separately (ie. we can reuse it elsewhere). So overall, some decent progress.

But some problems remain. If we really want the ability to swap out Audit systems, bundling the AuditFacade interface and AuditFacade1 implementation into the same module doesn’t give us the flexbility we need. While we can easily create a new AuditFacade implementation and allocate it to the audit.jar, this requires us to redeploy audit.jar unnecessarily. In the next post, we’ll explore how we can change the module structure to allow a new AuditFacade implementation while also allowing the existing AuditFacade1 implementation to be removed from the system.

@kirkk.com blog: Applied Modularity - Part 1

Mon, 16 Nov 2009 18:05:36 +0000

This is my first in a series of posts that shows how we can modularize Java applications today. A few notes before we get started. First, the examples are all core Java, with no OSGi. I’ll explain why in just a moment. Second, I’ve applied these techniques on real projects, so I know that they work.

Why No OSGi?

So why didn’t I use OSGi? Good question. One would think that if I’m going to modularize my system, I’d want to use a module framework. There are a couple of reason I chose not to.

While OSGi is a module system, OSGi will not help you design more modular software. Among other things, designing modular software requires that we understand the weight and granularity of individual modules, and use the right techniques to manage the coupling between modules. In other words, designing good software is our job. Tools and technologies may help, but make no guarantee. I spend more time talking about this in The Two Faces of Modularity & OSGi.
Most of us aren’t able to leverage OSGi today because the platforms and langauges we use don’t support it. I want to use the same tools and techniques that we can leverage in the enterprise right now. This is where the modularity patterns, as well as a few other tools, will help.

In general, I really want to focus on the design paradigm, not the tools and technologies. So instead of answering, “How do I use OSGi?”, I want to focus on “How do I modularize my system?”. Hopefully, that resonates with you. For the curious, if you do want to see the difference that OSGi makes, I have OSGi-ified the final version. But you’ll have to wait.

Let’s Get Started - The System

Interestingly, I’ve found that when designing modular software, it’s tough to identify the modules early in the lifecycle. Instead, shifts typically occur, and as things unfold, the modules become more apparent as development progresses. With a SOLID OO design, it’ll make it much easier to move things around and create new modules. So to start, while the system is small, I favor larger (coarser-grained and heavier-weight) modules.

As specific needs emerge, we’ll break larger modules out into a bunch of smaller (finer-grained and lighter-weight) modules that address specific needs (both functional and non-functional requirements). If you’re interested in the abstract essence of what I’m referring to here, you should read some of my prior blog posts (Agile Architecture might be a good place to start). It’s a deep and very interesting topic, and impacts how we understand software, maintain the system, reuse software entities, and more. Or, you can wait and see what I’m talking about, as we’ll experience this phenomenon as we move through the exercise.

Here’s a simple, high-level description of the system we’ll develop. It’s the common bill payment sample system often used.

We’ve been asked to develop a system to handle payment of of bills. Prior to paying the bill, the system should apply a discount to the bill in an amount that has been negotiated with the payee (we call this the process of auditing the bill). Applying this discount is a fairly complex process, and a 3rd party vendor has been commissioned that will apply this discount. Additionally, we must integrate with a legacy financials system that must be fed payment information for reconciliation.

We’ll flesh out additional details as development progresses.

Version One

The initial version for this system uses Struts as the web framework, and packages everything into a single WAR file. The initial class diagram can be seen at right (click to enlarge). It’s not a complete class diagram, but it does show the main abstractions. I’ve greatly simplified the system for purposes of example.

As you can see, there are some Action and ActionForm classes that leverage Struts. There are also a couple of JSP you don’t see here, but will if you take a look at the project. A Customer has a list of Bills, and each Bill has a reference to an AuditFacade and Payment class. The AuditFacade integrates with the 3rd party vendor software that applies the discount, and the Payment class integrates with the legacy financials system. Of particular interest, note the bi-directional relationship between Bill and these classes - a sure sign of a problem that will haunt us later. But don’t worry, we’ll fix it.

For the sake of simplicity, I’ve hardcoded the database into the data access layer, which is represented by the Loader interfaces. This way, you can experiment with the system without running any DDL scripts to create the database. If you really want to, feel free to add a real database backend. It wouldn’t be very difficult, but it’s not what I want to focus on here. The code for this initial version can be found in my Google Code Repository.

First Refactoring

So packaging everything up into a single WAR file for deployment certainly isn’t modular. In most systems we develop, we try to design layers that encapsulate specific behaviors and isolate certain types of change. Typical layers include a UI layer, a business or domain object layer, and a data access layer. In this system, we have these three layers. The Struts action and Form classes represent a part of the UI layer, which is shown in red. The Customer, Bill, Name, AuditFacade, and Payment form the business object layer, and the Loader classes form the data access layer. These classes are shown in blue. Now, here’s a key statement that you need to take with you.

If I truly have a layered system, then I should be able to break out each layer into a separate module where modules in the upper layers depend on modules in lower layers, but not vice versa.

If you try this, it’s likely you’ll find it’s not so easy. The takeaway here? Most development teams feel they have a layered architecture, but in reality, they don’t because somewhere deep within the bowels of the system lies an import or reference to a class higher up in the food chain that we aren’t aware of.

In fact, if I really do have a layered system, then I shouldn’t have to change anything other than my build script to break the layers out into separate JAR files. If I do have to change more than a build script, then I didn’t have a layered system to begin with, and I should perform some architectural refactoring to clean things up. Anyway, the end result is relatively simple to understand. No code changes. Only a build script change. And the structure shown at right is the result (click to enlarge). You can view the refactored project in the PhysicalLayers project, which includes the change to the build script on line 41.

Wrapping Up and Getting Ready for Part 2

This first refactoring was quite simple, but has significant implications. Foremost, it proves that my class level architecture was pretty decent. I was able to break the system out into modules for the various layers without having to change a bunch of code. Really, that’s the reason why it was so simple…because the design was decent. Had the design been shoddy, it would have been significantly more difficult pulling off this refactoring. Trust me!

Yet, as we’ll see, the existing design may meet the needs of today, but it’s going to have to evolve as change emerges. In part 2 of this series, we’ll take a look at what we need to do to integrate with another auditing system, and how modularity can help us do this. And as we progress, the amazing transformation of a system lacking modularity to a highly modularized version will unfold. Stay tuned!

@kirkk.com blog: OSGi Adoption Increasing?

Tue, 10 Nov 2009 19:31:46 +0000

Check out this graph obtained by searching for OSGi job trends on indeed. It’s evidence that interest in OSGi is increasing. That’s a pretty significant spike over the past two years. While it represents only a fraction of overall Java jobs, it’s refreshing to see that folks recognize the benefits of modularity (here’s an example). I can only expect that this graph will continue to trend upward over the next several months

@kirkk.com blog: Agile Architecture Deck & Code

Thu, 05 Nov 2009 19:04:44 +0000

Here’s my Agile Architecture - Patterns & Technology slide deck (bottom of this post) that I presented at SpringOne2GX and OOPSLA. Thank you to all who attended, and for providing such positive feedback. I’ve also uploaded the code samples used to show the modularity pattern examples found toward the end of the presentation. The code can be found under the billpayevolution project in my Google code repository.

A Bit More on the Code

The project is named billpay, and is a very simple system. It starts out in life as a project that’s bundled and deployed using a single WAR file (InitialVersion). Through a number of refactorings (7 in total), the project is broken out into separate modules, and the final version of the system (Refactoring7ImplementationFactory) has a number of modules (JAR files) that are included in the WEB-INF/lib of the WAR.

I intentionally avoided using OSGi for this exercise, primarily to show how we can modularize an application without a runtime module system. I do have an OSGi version of the system that can be deployed to an OSGi runtime (Tomcat using the Equinox Servlet Bridge is what I used), but I haven’t uploaded that code to the repository yet because I have a few things I need to change (like using Spring DM to remove the code dependencies on the OSGi API). That’ll come later, after I’ve had a chance to do a bit more work with it.

Each project along the path toward refactoring the application into modules comes with it’s own build script. For each project, except for the final version (Refactoring7ImplementationFactory), all you’ll need to execute the build is Ant and JUnit.

A Note on JarAnalyzer

Because I used JarAnalyzer in the final refactoring, you’ll need to make sure you have GraphViz installed and then modify the build script to point to the dot executable. Of course, you can also comment out the JarAnalyzer target, too. But JarAnalyzer was very helpful in identifying and managing the dependencies between modules, so I think you’ll find it useful to keep it. If you’re not familar with JarAnalyzer, it produces visual output showing the dependencies among JAR files (shown at right, click to enlarge), as well as an HTML report with some of the Martin metrics. Here’s the JarAnalyzer HTML report for the final version of billpay.

Like with any project, there are a few things I’d like to change. For example, in Refactoring7ImplementationFactory, I’d like to use Spring to inject the implementation classes instead of using the factory. Sometime soon, I hope to walk through these samples in a series of blog entries that shows how modularity can have an amazingly positive influence on the architecture of a software system. Until I do this, or you attend one of my presentations (btw, I’m speaking next week at Agile Development Practices), it may be a tad difficult to understand the forces behind my decisions (hint: they were driven by various business requirements). So please be patient, and stay tuned.

And Now the Presentation

Here’s the presentation, hosted on SlideShare. I look forward to your questions and/or any feedback you might have!

@kirkk.com blog: Turtles and Architecture

Tue, 03 Nov 2009 20:39:23 +0000

The story of software architecture reminds me of the following story.

“A well-known scientist (some say it was Bertrand Russell) once gave a public lecture on astronomy. He described how the earth orbits around the sun and how the sun, in turn, orbits around the center of a vast collection of stars called our galaxy. At the end of the lecture, a little old lady at the back of the room got up and said: “What you have told us is rubbish. The world is really a flat plate supported on the back of a giant tortoise.” The scientist gave a superior smile before replying, “What is the tortoise standing on?” “You’re very clever, young man, very clever”, said the old lady. “But it’s turtles all the way down!”

- A Brief History of Time by Stephen Hawking

Software architecture is “‘turtles all the way down.”

The Ivory Tower

Many of us can relate. In dysfunctional organizations, architects and developers fail to communicate effectively. The result is a lack of transparency and a lack of understanding by both sides, as shown in the diagrm (click to enlarge). The failure often occurs (though I recognize there are other causes) because architecture is about breadth and development is about depth. Each group has disparate views of software architecture, and while both are warranted, a gap between these views exists. The architect might focus on applications and services while the developer focuses on the code. Sadly, there is a lot in between that nobody is focused on. It is this gap between breadth and depth that contributes to ivory tower architecture.

Turtles and The Tower

Without question, the ivory tower is dysfunctional (regardless of cause), and systems lacking architectural integrity are a symptom of ivory tower architecture. So assuming good intent on the part of the architect and the developer, how can we bridge the gap between breadth and depth? How can we communicate more effectively? How do we increase understanding and transparency?

My favorite definition of software architecture was offered by Ralph Johnson in an article by Martin Fowler. He states:

In most successful software projects, the expert developers working on that project have a shared understanding of the system design. This shared understanding is called ‘architecture.’ This understanding includes how the system is divided into components and how the components interact through interfaces. These components are usually composed of smaller components, but the architecture only includes the components and interfaces that are understood by all the developers…Architecture is about the important stuff. Whatever that is.

The key aspect of this definition that differentiates it from many other definitions of architecture is that of “shared understanding.” We must have a shared understanding of how the system is divided into components, and how they interact. Architecture isn’t just some technical concept, but also a social construct. And it is through this that we can break down the divide between architects and developers.

To ensure shared understanding, we have to architect “all the way down.” Architects cannot worry only about services and developers cannot worry only about code. There is a huge middle ground that each must also focus on, as illustrated by the diagram at right (click to enlarge).

Focusing exclusively on top level abstractions is not enough. Emphasizing only code quality is not enough either. We must bridge the gap through other means, including module and package design. Often times, when I speak, I ask the audience to raise their hands if they spend time on service design. Most do. I also ask them to raise their hand if they spend time on class design and code quality. Again, most do. But then I ask if they also spend time on package and module design. Usually, only a small percentage leave their hands raised.

This is unfortunate, because module and package design are equally as important as service and class design. But somewhere along the way, with our emphasis on services and code quality, we’ve lost sight of what lies in between. Within each application or service awaits a rotting design, and atop even the most flexible code sits a suite of applications or services riddled with duplication and lack of understanding. A resilient package structure and corresponding software modules help bridge the divide between services and code.

Certainly, if I’ve said it once, I’ve said it at least a few times, we need to start focusing on modularity to ensure a consistent architecture story is told. It is the glue that binds. It’s the piece that helps bridge low level class design with higher level service design. It’s the piece that helps bring down the ivory tower, enhance communication, increase transparency, ensure understanding, and verify consistency at multiple levels. It is the piece that allows us to “architect all the way down.”

@kirkk.com blog: Question on Module Design

Fri, 30 Oct 2009 19:55:02 +0000

Like last year, in my Agile Architecture - Technologies and Patterns session at SpringOne2GX, I asked the attendees the same three questions surrounding class, package, and module design. This year, I had roughly 80 folks attend the session, and here is the rough breakdown of the hands shown after each of the questions.

How many spend time designing classes, both the behavior of a class and the relationships between classes? About 80% of attendees raised their hands.
How many spend time designing packages, both the behavior of a package and the relationship between packages? Roughly 20% raised their hands.
How many spend time designing JAR files, both the behavior of a JAR and the relationship between JAR files? Again, about 20% raised their hands.

These are consistent responses to what I see elsewhere, as well. I was hopeful that since I was attending the Spring conference, a few more developers were leveraging OSGi and actually spending some time on module design. But that doesn’t look to be the case. Maybe modularity isn’t sexy enough? I suppose that’s just a bit more fodder for the argument that there is no migration path for modularity, and that we need better tools, tutorials, and educational materials to help us design modular software. Fact is, I had more than one person stop to ask me where they can find more information. And that’s why recently, I’ve been focusing my talks on what we can do today, right now, to design more modular software.

I know we’ll be there someday. After my OOPSLA tutorial this week, Alex Buckley stopped by and we chatted for a while on modularity in JDK 7. Whether it’s Jigsaw, OSGi, or both, we can’t be sure. But one thing is certain - modularity is coming to the Java platform, and while it might not be all that cool and exciting right now, it’s going to play a significant role in how we architect and design applications going forward.

@kirkk.com blog: OSGi Survey Results

Thu, 22 Oct 2009 14:50:36 +0000

Update (10/23/09): There’s been some interest by folks in seeing the details for the free form questions (Question #5 and Question #10). Here’s the raw data, exported directly from the survey. These are each PDF documents.

Download response to Question #5 on OSGi frameworks currently being used and evaluated and download response to Question #10 on obstacles OSGi must overcome.

I’ve also opened sharing of the survey, so you can see first-hand the survey results.

Last week, I opened the 2nd Annual OSGi Survey. The survey concluded early Tuesday, with more than 250 developers taking the survey (last year, 64 took the survey). The survey was not scientific, and it was clear that the majority of folks taking the survey were already using OSGi. But the results were interesting nonetheless, and offers some insight to how OSGi is perceived and used. Here are the results!

The Questions and Answers

Let’s analyze the results, question by question. Then, we’ll offer up some analysis, with possibly more to come later. If you’re interested in last year’s results, I posted a summary of the results on my site with some detail on the APS Burton Group blog.

Question #1 - Are you familiar with the OSGi Service Platform?
92% (241) have heard of OSGi, as compared to 89.1% (57) last year.

Question #2 - Are you currently developing systems to be deployed to an OSGi runtime?
70.9% (185) were using OSGi as compared to 50% (32) last year.

Question #3 - Do you have plans to develop systems that are to be deployed to an OSGi runtime within the next 6 – 12 months?
82.7% (215) are planning to use it in the next 6 - 12 months as compared to 79.7% (51) last year.

Question #4 - Are you familiar with the available OSGi products and services?
83.5% (217) were familar with OSGi products and services as compared to 67.2% (43) last year.

Question #5 - Which OSGi implementations are you currently using or evaluating?
This was a freeform question, and in both categories (Using and Evaluating), Equinox was the clear leader, garnering more than 100 responses, with Felix coming in second. Other notables mentioned include dm Server, ProSyst mBedded Server, and Paremus Service Fabric.

Question #6 - Which of the following types of systems are you planning to deploy within an OSGi runtime?
Respondents were able to select more than one answer, resulting in a total that exceeds the number of folks taking the survey. So it’s obvious we are leveraging OSGi for different types of applications (click image for summary). In total 64.8% (162) (65.1% last year) said they were developing web apps, 17.2% (41) (11.1 % last year) were using it for embedded development, 15.6% (39) (12.7% last year) on mobile development, 50.4% (126) (39.7% last year) for rich clients, and 13.6% (34) (17.5% last year) said they weren’t using OSGi at all.

Question # 7 - What are the benefits you hope to achieve using OSGi?
Again, respondents were able to select more than one answer for this question (click image for summary). In total, 83.9% (214) (82.5% last year) wanted greater modularity, 56.6% (144) (63.5% last year) wanted hot deployment, 51.4% (131) (54% last year) wanted a more manageable runtime environment, 68.6% (175) (71.4% from last year) wanted better dependency management, 57.3% (146) (63.5% from last year) wanted bundle versioning, 63.5% (162) (65.1% last year) wanted a plugin architecture, while 10.2% (26) (14.3% from last year) abstained.

Question #8 - What are your greatest technical challenges in using OSGi?
The clear challenge in using OSGi (click image for summary), cited by 63.8% (157) (63.5% last year) of respondents, is how to use OSGi most effectively. Following a rather distant second place at 40.2% (99) (38.1% last year) was finding valid OSGi bundles. It tailed off pretty sharply after that with testing at 30.9% (76) (19% last year), debugging at 22.4% (55) (25.4% last year), and the remaining coming in each close to 11%. These were each down from last year.

Question #9 - If your application server provided an OSGi runtime container, would you consider deploying your software as modules/bundles?
87.5% (223) (88.7% last year) said they would deploy their software as modules if their platform supported it. 8.2% (21) (9.75% last year) said no they would not. 4.3% (11) (1.6% last year) said the weren’t sure what an OSGi bundle was. In total, 7 people skipped the question.

Question #10 - What obstacles must OSGi overcome to achieve widespread adoption within the enterprise?
Since this was a free form text entry field, I took the liberty of classifying the various responses. While not everyone chose to offer input, for those who did, the following obstacles were cited by the respondents as being most significant:

51 cited lack of tooling
28 cited a migration path from Java EE to OSGi, including a lack of platform support as an issue
23 cited a lack of good education material
10 cited lack of available bundles

There were myriad other responses, as well. Some stated that complexity was an obstacle, which Hal Hildebrand refers to as the “cognitive burden” of OSGi. Another respondent pointed out that the availability of an OSGi-based “killer application” that wasn’t an IDE would also increase the visibility of OSGi. Yet another respondent pointed out that OSGi is an over-engineered solution that actually causes more problems (due to its complexity) than it solves.

Closing Thoughts

What might be most interesting is that there is little significant difference from last year’s results. The results hover around the same numbers as last year, with only Question #2 and Question #4 seeing the most significance differences. There were a few aspects of the survey that struck me as particularly interesting. Tooling, a migration path, and educational material surrounding how to use OSGi most effectively continues to be a significant issue for developers, even though roughly 15% more of this year’s respondents claimed familiarity with OSGi products and tools. In other words, they’re familiar with the technology, but still recognize the need for better tools, etc. Clearly this is supported given the responses to Question #8, where the greatest challenge in using OSGi is understanding how to use it most effectively.

Last year, 50% said they were using OSGi while close to 80% said they planned to use it in the next 6 - 12 months. The response to Question #2 indicates we saw a 20% growth in adoption…somewhat close to the 30% indicated.

Developers again indicated that they want their application platforms to expose OSGi for building web applications, with almost 90% saying they’d develop web applications today using OSGi if the platform allowed them too.

Finally, the majority of folks currently using or evaluating OSGi are doing so at the framework level, not the platform level. Given that most people are planning to develop web applications using OSGi, I really would have thought that platform use and evaluation (such as dm Server or Infiniflow) would have been higher. It’s possible the way I worded Question #5 influenced how respondents answered the question.

Platform support for OSGi is coming. Well, at the very least, platform support for modularity is coming. However, once the platform does support modularity, and a rich suite of tools is available, there is still the nasty little problem surrounding how we effectively design modular software. It appears we’re on the right track, but we still have a ways to go.

@kirkk.com blog: 2nd Annual OSGi Survey

Wed, 14 Oct 2009 17:59:38 +0000

Calling all Java Developers, please take the 2nd Annual OSGi Survey!

Last year, I conducted a survey to gauge industry acceptance, adoption, and understanding of OSGi. I posted a summary of the results on my site with more detail on the APS Burton Group blog. With SpringOne2GX approaching in less than one week, I decided to offer the same survey to get a feel where OSGi has gone over the past year. I contemplated changing a few of the questions to reflect what I thought would be more relevant this year, but decided against it because I really wanted an apples-to-apples comparison. The survey certainly isn’t scientific, but it’ll sure offer some decent (and maybe even interesting) insight to the OSGi market.

The Survey consists of 10 simple questions, and should take only a few minutes of your time. So if you’re involved with development on the Java platform, and can spare a few moments, your time would be greatly appreciated. I plan to stop collecting responses on Monday, October 19th, and hope to share the results in my session at SpringOne2GX next week titled Agile Architecture - Technologies and Patterns, as well as sharing the details in a blog post.

Update: Please note the DZone link that was submitted states the survey will close on Monday, Nov. 20th. That’s incorrect. The survey will close on Monday, October 19th.

@kirkk.com blog: The Use/Reuse Paradox

Wed, 07 Oct 2009 17:03:32 +0000

There are certain paradoxes that create conflict when designing software systems. The paradoxes result in opposing forces that are counterintuitive, and require further examination to more fully understand how the tension can be resolved. Here, I explore the tension between use and reuse. Certainly there are others, too. What software development paradoxes have you encountered?

Use and Reuse

I have a simple question. What’s the difference between “use” and “reuse”? Dirk Riehle broaches the subject in suggesting that using a component is when you embed that component in a collective work and reusing a component is when you create a derivative of that component. I use the term “component” above, but it could just as easily be replaced by “method”, “object”, “module”, “service”, or anything else that you want to use. Or should I say reuse? Funny!

A few responses to the same question I posted to Twitter garnered some responses from @RSessions, @bigballofmud (Brian Foote), and @IsaGoksu, though not enough to offer perfect clarity.

The topic can be quite confusing. For instance, is invoking a once deployed service from multiple consumers use or reuse? And how does this differ from including a component in multiple services? Which is use and which is reuse? The differences between these two scenarios can be seen in the diagram at right (click to enlarge).

A Trick Question

Instead of trying to distinguish between reuse and use, let’s consider an alternative perspective. If we adopt a canonical definition of reuse, we can state that it means to simply “leverage an existing asset”. Now, let’s define use as “the ability to leverage an asset”. If we’re willing to accept these definitions, then the relationship is opposing - as one goes up, the other goes down. A module or service might be highly reusable, but very difficult to use. Likewise, a module or service might be very easy to use, but difficult to reuse. And it’s incredibly difficult to offer both.

If we make a software entity highly reusable then it’s likely a lightweight and fine-grained entity. This allows for environmental configuration driven by context and extensibility through well-defined interfaces and extension points. But with this flexibility is additional complexity that makes the entity inherently more difficult to use (flexibility and complexity are another paradox?). I explore these ideas further in Reuse: Is the Dream Dead, and draw the conclusion that

Maximizing reuse complicates use.

Dealing With It

Understanding the forces at play here is important because they are consequential to architecture, and resolving the tension is an aspect of architectural agility. It’s virtually impossible to design a reusable software entity until we have a better understanding of it’s usage scenarios (I recall a “rule of 3″, but can’t seem to place it. Anyone know?). And since the unit of reuse is the unit of release, it figures that modularity plays a prominent role here.

Understanding principles, patterns and practices (like SOLID and modularity patterns) that increase architectural agility help resolve the tension between use and reuse, and are certainly a step in the right direction. But so too is understanding that certain decisions must be deferred until we have the requisite knowledge to make the most informed decision possible. Because of this, we should strive to minimize the architectural significance (impact and cost) of change by making our designs as reversible as possible. Reversibility doesn’t always mean great flexibility, though. Sometimes it means we make something as simple as possible so that it’s easy to change later. Either way, it’s imperative to accommodate the natural shifts that occur throughout development, and modularity plays a central role in making this happen.

Moving On

For those that follow this blog, you’ll know it’s not the first time I’ve written about this topic. For others, if you’re clicking on any of the links in this post, you’re quickly discovering that, as well. Going forward, I intend to explore many of these concepts using some concrete examples that should offer a bit more insight to the discussions. I’ve put together some sample exercises for some upcoming conferences, and I intend to walk through those samples in a series of future posts. The result will be roughly seven or eight separate posts that show the evolution of a system. There’ll be code, builds, tests, and of course, modularity. Along with a lot of other stuff, too.

For now, if you’re interested in this topic, as well as ways to increase architectural agility, you might consider checking out some of my following entries (some of which are linked to above) related to this topic. You can bet there will be more coming, too!

Modularity by Example - A simple visual example illustrating the benefits of modularity.
Agile Architecture, Lean Principles - Comparing my past thoughts on agile architecture to the Lean Principles of Software Development.
Modularity & Architecture - A response to the entry on eliminating architecture.
Eliminate Architecture - Discusses the goal of architecture and how to eliminate the impact and cost of change.
Agile Architecture - My views on agile architecture and the natural architectural shifts that occur throughout the development lifecycle.
Agile Architecture Requires Modularity - Discusses the role of modularity in agile architecture.
On SOLID Principles and Modularity - Discusses where you need flexibility in architecture.
Two Faces of Modularity and OSGi - Introduces the need for patterns and tools to help design more flexible and modular architecture.
Reuse: Is the Dream Dead - Discusses the tension between reuse and use.
Modularity Patterns - Presents 19 modularity patterns that help ease the tension between
reuse and use while making a software system easier to understand,
maintain, and extend.

@kirkk.com blog: Lean and Kanban Collection

Fri, 02 Oct 2009 14:48:43 +0000

For those of you interested in Lean and Kanban, but not much knowledge yet, it can be tough filter the noise and find information explaining what they are and how they might be different from what you’re already doing (agile, Scrum, etc.). Since top 10 lists are rather passé, I’ve gone the whole nine yards and scoured the blogosphere to come up with the best nine posts I could find that capture the essence of Lean and Kanban.

The list is in no particular order, and there’s no doubt that I’ve missed some great posts that offer insight. The blogosphere is a pretty big place, after all. Naturally, I’ve cheated a bit by pointing to related posts from within the top nine. Enjoy!

The Essence of Lean and Kanban - Post by @alshalloway that talks about Flow.
Patterns and Practices of Lean Software Development - Post by @corey_ladas discussing principles such as One Piece Flow, Real Options, and Build Quality In.
Noodling on Kanban - The first of a four-part series written by @mcottmeyer. Also be sure to check out Part 2, Part 3, and Part 4.
How Is Kanban Different From Other Approaches - Post by Karl Scotland that examines Kanban’s primary practices. Also check out his follow-up post where he adds The Fifth Primary Practice of Kanban. And his posts on Kanban, Flow, and Cadence and XP as a Kanban System are great, as well.
Kanban Applied to Software Development: From Agile to Lean - A detailed article on InfoQ that begins by exploring the origins of Kanban in the Toyota Production System, before launching into a discussion on Kanban in software development.
Kanban in Action - An early post by @agilemanager (David Anderson) where he describes his initial forays into Kanban.
I Have Heard of Kanban - A response to the critical post, Have you heard of Kanban, by Matthew Huesser. There were actually a few posts that were part of this conversation. It started with @chris_mcmahon calling into question Kanban with his post, against kanban. That led to Matthew’s post, and ultimately the response. Matthew has since posted a Kanban Redux and Chris has posted a couple of responses in against kanban part 2 and against kanban part 3.
Bumping into Scrum’s Boundaries - Part 1 of @damonpoole’s Transitioning from Scrum to Kanban Series. Part 2 can be found in his post on One Piece Flow.
The Poppendieck’s on Lean - An interview with Mary and Tom Poppendieck on Lean.

If you’re interested in learning more about Kanban, consider joining the Kanban Development Yahoo Group. Additionally, the Limited WIP Society is the home of Kanban Development, and contains some good definitions of what Kanban means within software development. And of course, the best place to learn more about Lean is to start with the Poppendiecks.

Note: Graphic courtesy of InfoQ. Used without permission.

@kirkk.com blog: Duct Tape Programming

Fri, 25 Sep 2009 14:50:13 +0000

Joel tells the story of The Duct Tape Programmer, and Uncle Bob offers his response. Now these are two pretty smart guys who know a lot about software development. But when we receive fundamentally different messages from a couple of industry luminaries like Joel and Uncle Bob, we’re left wondering - “Who is right? Who should we listen to?”

Joel makes some valid points, but it’s strange that immediately after discussing the dangers of overengineering and the importance of shipping the product, he points out the following:

Zawinski didn’t do many unit tests. They “sound great in principle. Given a leisurely development pace, that’s certainly the way to go. But when you’re looking at, ‘We’ve got to go from zero to done in six weeks,’ well, I can’t do that unless I cut something out. And what I’m going to cut out is the stuff that’s not absolutely critical. And unit tests are not critical. If there’s no unit test the customer isn’t going to complain about that.”

I’m not quite sure how unit testing encourages overengineering nor how unit testing impedes shipping the product. Unit testing does neither. In fact, unit testing inspires a simpler design, not a more complex one. And because unit testing helps developers build quality into the product, I’m certain it doesn’t inhibit shipping the product - or at least a product that works. The correlation makes no sense, and I can’t help but wonder if the message wasn’t an intentional jab based on some past conflicts.

Of course, I certainly agree that favoring simplicity and getting the job done are righteous goals. Sometimes you have to cut a few corners, take a few shortcuts, and implement a solution that isn’t perfect. But there still exist fundamental practices of professionalism that must serve as our guide. And as Uncle Bob points out, unit testing is one of these practices. Bottom line! As a humorous post on twitter suggests:

never take software advice from a bug tracking system salesman

Unit testing is important, has significant long-term benefits, and developers should use them. However, I do take slight issue with Uncle Bob’s following statement:

I found myself annoyed at Joel’s notion that most programmers aren’t smart enough to use templates, design patterns, multi-threading, COM, etc. I don’t think that’s the case. I think that any programmer that’s not smart enough to use tools like that is probably not smart enough to be a programmer period.

In this respect, Joel is right. There might be a lot of people out there who aren’t smart enough to be programmers, but the reality is that only 50% of us can be above average. Of course, we all think we’re in the upper 50 percentile and when we’re looking at code written by someone else, it’s apparent to us it’s the other guy or gal who’s in the lower half. Without arguing who belongs where, the reality is that if we were all smart enough to use these advanced constructs, we wouldn’t be left with the mess we have today.

But really this only reinforces the importance of unit testing, which is why statements that denounce unit testing are so surprising. Think about it…Would you rather maintain a codebase with near 100% test coverage or a codebase with near 0% test coverage? Maybe those who can’t answer this question are the folks that shouldn’t be programmers!

@kirkk.com blog: Agile Architecture, Lean Principles

Tue, 22 Sep 2009 19:44:53 +0000

Most of my discussions surrounding agile architecture have been focused on exploring how modularity helps increase architectural agility. I claim that modularity is a required (and to this point, missing) aspect of agile architecture. The basis for this claim follows:

Architecture is design, but not all design is architecture (ala Booch).
Design is architecture if the design is architecturally significant. That is, if it’s hard to change.
The goal of architecture is to eliminate the impact and cost of change.
The way to eliminate the impact and cost of change is through flexibility.
With flexibility comes complexity. We must therefore strive to increase flexibility while taming complexity.
Modularity helps us identify where we need flexibility by understanding the joints of the system where flexibility is necessary.

Without modularity, we can’t identify the joints so it’s more difficult to understand where we need the flexibility. My posts titled Modularity & Architecture and Modularity by Example show visual examples comparing designs that are isolated and insulated to designs that span the joints of a system. I’ve also devoted extensive discussion to these ideas in a number of other posts, which I summarize in my Agile Architecture Presentation post.

Lean Principles

Recently, I’ve been spending more time exploring lean software development principles and their relationship to agile architecture, and the best place to start when examining lean is with the Poppendiecks. I’m fascinated by the synergy that exists between the lean principles and agile architecture. In fact, when reading the second chapter of their book, “Implementing Lean Software Development: From Concept to Cash“, I was pleasantly surprised by an interesting discovery.

It seems that a study of software development practices by Harvard Business School professor Alan MacCormack revealed four fundamental practices that lead to successful software development. These include releasing early, continuous integration, experience and instinct, and a modular architecture. So it seems I’m not alone in feeling modularity is a critical component of agile architecture. But the thrust of the discussion comes later on in Chapter Two, when speaking of deferring commitment.

Deferring Committment, Reversibility, and Eliminating Architecture

Deferring commitment focuses on two fundamental factors - reversibility and irreversibility. In general, reversible decisions are those that can be changed while irreversible decisions are those that cannot be changed. We should strive to make irreversible decisions at the last responsible moment. For it is at this moment when we possess the most knowledge that will allow us to choose the most viable option. But we are also advised that, and I quote:

“First and foremost, we should try to make most decisions reversible, so they can be made and then easily changed.”

For me, this captures the essence of eliminating architecture. If we are able to take a seemingly architecturally significant challenge and make it reversible, then we have effectively minimized the impact and cost of change to a point where change is no longer architecturally significant.

Going forward, I intend to more fully explore additional synergies between lean software development principles and agile architecture.

@kirkk.com blog: Modularity & Architecture

Wed, 16 Sep 2009 16:36:20 +0000

I recently wrote about eliminating architecture, and there were a few comments, especially by folks on JavaLobby, who thought I had my head in the clouds. Too much theory. Too many abstract concepts. Not achievable in a real world development scenario. That I’m making a play on words. Let’s take another angle.

Defining Architecture

There are numerous definitions of architecture. But within each lies a common theme, and some key phrases. Here are a few of the definitions. From Booch, Rumbaugh, and Jacobson in UML User Guide (Addison-Wesley, 1999):

An architecture is the set of significant decisions about the organization of a software system, the selection of the structural elements and their interfaces by which the system is composed, together with their behavior as specified in the collaborations among those elements, the composition of these structural elements and behavioral elements into progressively larger subsystems, and the architecture style that guides this organization — these elements and their interfaces, their collaborations, and their composition.

From the ANSI/IEEE Std 1471-2000:

The fundamental organization of a system, embodied in its components, their relationships to each other and the environment, and the principles governing its design and evolution.

In TOGAF, architecture has two meanings depending on context:

1.) A formal description of a system, or a detailed plan of the system at component level to guide its implementation
2.) The structure of components, their inter-relationships, and the principles and guidelines governing their design and evolution over time.

And in a QCon presentation by James Coplien and Kevlin Henney, architecture is defined as:

1.) Architecture embodies the critical design decisions that typify a system. Relates to cost of change, organizational structure, structure of code, capabilities of a system, etc.

2.) The significance of decisions needs to be understood and assessed. A heavy-weight approach is likely to reduce understanding and our ability to assess

Certainly, we see a pattern here as some key phrases and terms recur. These include “significant/critical decisions”, “components”, “structure”, and “cost of change”. And the definition from the Fowler article (which is actually part of a statement made by Ralph Johnson) introduced in the Eliminate Architecture post uses similar terminology and phrases.

It is these definitions that led us to the goal of architecture - eliminate the cost and impact of change. If we are able to eliminate the cost and impact of change, then change is no longer architecturally significant. If something isn’t architecturally significant, then we don’t consider it architecture. Essentially, we’ve eliminated architecture because the cost of change is no longer significant. That must be what we strive to achieve. And the way to achieve this is by increasing flexibility while taming complexity. Let’s take an analogy.

An OO Analogy

The goal of object-oriented design can be summed pretty effectively using the open-closed principle, which states:

“a system should be open for extension but closed to modification.”

This is done through abstraction and inheritance. If we have abstractions in the right place, we can extend the system by introducing new classes without modifying existing system classes. The key is that we have to recognize which areas of the system require this added flexibility.

Certainly not all areas of the system can possess this flexibility. It’s unrealistic and entirely too complex. But it should still be our goal. It’s the same with architecture. Obviously we cannot eliminate the architectural significance of all change, but it must still be our goal. Otherwise, what goal are we striving to achieve?

Modularity - The Missing Ingredient

Two of the key elements of the architectural definitions are component and composition. Yet there is no standard and agreed upon definition of component (reminding me of architecture, actually), and most use the term pretty loosely to mean just “a chunk of code”. But that doesn’t work, and in the context of OSGi, it’s clear that a module is a software component. That’s excellent fodder for my claim that agile architecture requires modularity because agile architecture demands that we design a flexible system that allows us to make temporal decisions based on shifts that occur throughout development. Modularity has been a missing piece that allows us to more easily accommodate these shifts.

I illustrate this using the simple diagram at right, and it’s what I was referring to in the Eliminate Architecture post when I stated that modularity, in conjunction with design patterns and SOLID principles, represent our best hope to minimize the impact and cost of change. It’s easier to change a design embedded within a module than it is a design that spans modules. I provided a similar example in Modularity by Example.

In other words, it’s easier to isolate change by insulating a design within a module. Designs that span modules are the joints of our system, and changes in these areas are more complex and costly. This is where we need the flexibility. It’s where we need stability. But if we don’t have modularity, we don’t have joints. Or maybe everything becomes a joint? Either way, without modularity we can’t identify the joints so it’s more difficult to identify where we need the flexibility. If we are able to avoid changes that span joints, that change is isolated…insulated…encapsulated within a module, and the impact of change is minimized..

Can we do this in all cases? Of course not! Just like we cannot always design software that’s open for extension but closed to modification. But it must be what we strive to achieve. Again, modularity has been the missing ingredient. And now is a perfect time to start understanding how to design more modular software.

@kirkk.com blog: Agile Architecture Presentation

Fri, 11 Sep 2009 18:00:49 +0000

The slide deck for my upcoming presentations at SpringOne 2GX, Agile Development Practices, and OOPSLA is complete. This presentation doesn’t spend much time talking about the “softer” side of agile architecture. I don’t make silly statements about avoiding the ivory tower, delivering solutions that work, or embracing change. These are given! I intentionally avoid the buzzwords, like Emergent Architecture, so you won’t find a session that’s full of fluff and short on substance.

I won’t present earth shattering principles of agile architecture. If you’re not already convinced that ivory tower architecture doesn’t work, that you need to find a way to embrace change throughout the lifecycle, or that delivering working solutions is the best way to move, then this may not be the session for you. The general thesis of the presentation follows.

The goal of architecture is to minimize the impact and cost of change. To do this requires we create flexible systems with the ability to adapt and evolve. But with flexibility comes complexity, presenting architects with a paradox. As we design more adaptable systems, the additional complexity reduces their ability to survive. Eventually, the software rots, the architecture crumbles, and the system fails. Therefore, we must find a way to increase adaptability without decreasing survivability.

The presentation begins by examining this goal of software architecture, helping explain why it’s so critically important that we accommodate the natural shifts in architecture that inevitably take place as a software system grows. In other words, change impacts architecture and this demands that we be able to accommodate architectural shifts throughout the development lifecycle. The meat of the presentation is spent exploring how to do this, with lots of real world examples interwoven throughout the discussion. In fact, the presentation is a partial case study that’s gleaned from about a decade of experience practicing what I’m preaching. For more details on these ideas, and the type of discussions we’ll have as part of the presentation, see a few of the following blog posts:

Eliminate Architecture - Discusses the goal of architecture and how to eliminate the impact and cost of change.
Agile Architecture - My views on agile architecture and the natural architectural shifts that occur throughout the development lifecycle.
Agile Architecture Requires Modularity - Discusses the role of modularity in agile architecture.
On SOLID Principles and Modularity - Discusses where you need flexibility in architecture.
Two Faces of Modularity and OSGi - Introduces the need for patterns and tools to help design more flexible and modular architecture.
Reuse: Is the Dream Dead - Discusses the tension between reuse and use.
Modularity Patterns - Presents 19 modularity patterns that help ease the tension between reuse and use while making a software system easier to understand, maintain, and extend.

Admittedly, the material I’m presenting isn’t all of my own creation. I view the session as a consolidation of the tried and proven. For example, Ralph Johnson helped backed my thesis when providing the following quote:

…making everything easy to change makes the entire system very complex…

I cite the SOLID principles. I spend some a great deal of time examining the role of modularity in agile architecture, which in my opinion, is a requirement (not the only one, but one that has been missing). I might even talk a bit about services. In the end, I pull all of this information together into what is a clear and cohesive strategy for increasing architectural agility. If you’re attending one of these conferences, I hope you’ll stop by.

@kirkk.com blog: Eliminate Architecture

Tue, 08 Sep 2009 17:58:28 +0000

I believe:

The best way to deal with architecture is to eliminate it.

Architecture’s Goal

Let’s start at the beginning. First, by defining architecture. Second, by understanding the goal of architecture. I’ll offer two perspectives, one from Fowler and the other from Booch. First, the Fowler statement.

In most successful software projects, the expert developers working on that project have a shared understanding of the system design. This shared understanding is called ‘architecture.’ This understanding includes how the system is divided into components and how the components interact through interfaces. These components are usually composed of smaller components, but the architecture only includes the components and interfaces that are understood by all the developers…Architecture is about the important stuff. Whatever that is.

Now Booch says:

All architecture is design but not all design is architecture. Architecture represents the significant design decisions that shape a system, where significant is measured by cost of change.

Examining these two statements is very revealing. Fowler makes it’s apparent that architecture is about the important stuff, while Booch is clear that something is architecturally significant if it’s difficult to change. Therefore, I’ll conclude that the goal of software architecture must be to eliminate the impact and cost of change, thereby eliminating architectural significance. And if we can do that, we have eliminated the architecture.

Eliminating Architecture

The Paradox

The idea behind eliminating architecture isn’t new. In fact, Fowler mentions “getting rid of software architecture” in the article linked to above. And the way to eliminate architecture by minimizing the impact of cost and change is through flexibility. The more flexible the system, the more likely that the system can adapt and evolve as necessary. But herein lies the paradox, and I’ll use a statement by Ralph Johnson to present and support the idea.

…making everything easy to change makes the entire system very complex…

So as flexibility increases, so too does the complexity. And complexity is the beast we are trying to tame because complex things are more difficult to deal with than simple things. It’s a battle for which there is no clear path to victory, for sure. But what if we were able to tame complexity while increasing flexibility? Is it even possible? In other words, how do we eliminate architecture?

Maximize Flexibility, Manage Complexity

Eliminating Architecture

As the Johnson quote clearly points out, it’s not feasible to design (or should I say architect?) an infinitely flexible system. Therefore, it’s imperative that we recognize where flexibility is necessary to reduce the impact and cost of change. The challenge is that we don’t always know early in the project what might eventually change, so it’s impossible to create a flexible solution to something we can’t know about. This is the problem with Big Architecture Up Front (BAUF), and it’s why we must make architectural decisions temporally. It’s also why we must take great care in insulating and isolating decisions we’re unsure of, and ensuring that these initial decisions are easy to change as answers to the unknown emerge. For this, modularity is a missing component that helps minimize the impact and cost of change, and it’s why agile architecture requires modularity.

In the UML User Guide (P. 163), Booch talks about “modeling the seams in a system.” He states:

Identifying the seams in a system involves identifying clear lines of demarcation in your architecture. On either side of those lines, you’ll find components that may change independently, without affecting the components on the other side, as long as the components on both sides conform to the contract specified by that interface.

Where Booch talks about components, today we talk about modules. Where Booch talks about seams, I talk about joints. Modularity combined with design patterns and SOLID principles, represent our best hope to minimize the impact and cost of change, thereby eliminating architecture.

@kirkk.com blog: Modularity Pattern - Manage Relationships

Wed, 02 Sep 2009 15:31:01 +0000

Recently, I posted an entry introducing 19 patterns for modularity. The first of these patterns was ManageRelationships. ManageRelationships is a simple pattern stating that we should design module relationships. I categorize this as a base pattern, meaning it’s a prerequisite pattern for many of the other patterns in the list. For example, the Dependency Patterns listed demand that we manage the relationships between modules. It’s a simple pattern, but important. The discussion below isn’t exhaustive, but it does offer a bit more insight to ManageRelationships.

Description

A relationship between two modules exists when a class within one module depends upon a class within another module. In other words,

If changing the contents of a module, M2, may impact the contents of another module, M1, we can say that M1 has a dependency on M2.

Dependencies can manifest themselves in different ways. The most straightforward is a direct dependency, where a client module depends directly on a service module (shown at left in the diagram). Indirect, or transitive, dependencies involve at least three modules, where one service module is also a client module. Also shown at left, the service module is a client module of the subsystem module and a service module to the client module. Here, the client module has an indirect relationship to the subsystem module, and if something changes in the subsystem module, it may ripple through the other modules.

Motivation

Understanding the relationships between modules makes it easier to isolate the impact of change to a specific set of modules, which is not something easily done at the class level. I provided an example of this in a previous post. In the example above, changes to the client module clearly indicate that the impact of change is isolated only to the client module. Likewise, changes to the service module indicate the impact of change could spread to other classes within the service module as well as classes within the client module. Without designing and understanding module relationships, it’s difficult to understand the impact of change.

Consequences

There are a lot of things to consider when designing module relationships. In general, a module has incoming dependencies, outgoing dependencies, or a combination of each. Different forces affect modules depending on the types of dependencies they possess.

Modules with a lot of incoming dependencies are more difficult to change because they are being reused by more client modules. Because of this, it’s imperative that they undergo more thorough and rigid testing, and steps should be taken to minimize changes to these modules (like using AbstractModules).

Modules with a lot of outgoing dependencies are easier to change because they are reused by fewer modules. Unfortunately, these modules are more difficult to test in isolation because of their dependencies on other modules.

There are ways to alleviate each of these challenges. Designing abstract modules, ensuring module relationships are acyclic, and separating abstractions from the classes that realize them are each examples.

Implementation

When designing module relationships, there are some important implementation details to consider. For example, a service module must be included in the build classpath of the client module. Failing to do so will result in a compile error. If the client module references a class in the service module at runtime, then the service module must also be included in the runtime classpath, as well. Failing to do so will result in a ClassNotFoundException.

Modules with excessive incoming and outgoing dependencies are the most difficult to manage because they are widely reused but also difficult to change and test. Because of this, it’s ideal if modules are either heavily depended upon or heavily depend upon another module. Unfortunately, this isn’t always possible. In these cases, other modularity patterns can help ease the tension when modules have a lot of incoming and outgoing dependencies.

Wrapping Up

Identifying the modules is the first step in designing modular software systems. Shortly following, however, is managing the relationships between those modules. It’s these relationships between modules that I refer to as the joints of the system, and it’s these areas of the system that demand the most nurturing to ensure a flexible system design. If care isn’t given to managing the relationships between modules, separating out the system into a set of modules isn’t going to provide significant advantages.

More on some of the other modularity patterns coming. In the meantime, your feedback is appreciated.

@kirkk.com blog: Conclusion of the Stars

Fri, 28 Aug 2009 19:30:46 +0000

Yesterday’s conclusion of Programming with the Stars at Agile 2009 was an awesome event. A huge thank you to Jeff Nielsen and Joshua Kerievsky for coordinating the event and pulling together the great list of judges and contestants. A huge thank you to the judges and contestants, as well. All made quite a commitment to the event. The final day of competition was the free style day, which meant the contestants could choose their exercise, and were given six minutes to strut their stuff.

The first pair of James Grenning and Kenrick Chien opted for a learning day. They provided a great overview of how a pair might learn from each other through some simple exercises. On numerous occasions, they jumped at the opportunity to suck up to the judges. The highlight came when Grenning, who works on a Mac, chose to jump into a virtual machine running Windows in an attempt to come under the good graces of Judge Newkirk who works at Microsoft, by stating:

Finally an opportunity to work in a Microsoft environment

Judge Marick, however, noted the importance of this playful attempt to suck up when he commented on the important role that VMs will have on the development environments and how developers will leverage them in developing software going forward.

Next up was Gerard Meszaros and Ola Ellnestam, who put on an excellent display of pairing. In the final judging session, Judge Marick jokingly expressed his concern over the obvious corrupt nature of the judges. The final judge voting gave the Meszaros pair a slight lead over Grenning and Chien. But the competition wasn’t complete.

To conclude, the audience got a vote, and while the votes were being tallied, a bonus round commenced. Each pair was required to disable their mouse and using only keystrokes, create a program that printed “Hello Programming with the Stars” to the console. While it appeared Grenning and Chien finished their coding exercise first, it took them a bit longer to execute the program. Possibly because C was their language of choice? Maybe, maybe not. Judge Marick noted that while the two contestants were completing the bonus round, he and Judge Newkirk tried the same exercise using Ruby, but that it took them longer than either of the two groups of contestants. He explained this pretty clearly by joking,

It’s clear that we’ve now proven what statically typed language advocates have been arguing in favor of for quite some time. Java and C are much better languages for mission critical applications.

In the end, it was a great week for Programming with the Stars, which was ultimately won by the Meszaros and Ellnestam pair. Here’s hoping that Agile 2010 serves as host to this fun competition!

For additional Agile 2009 coverage, see Stars and Craftsmanship and Agile 2009 - Day 3.

@kirkk.com blog: Agile 2009 - Day 3

Thu, 27 Aug 2009 06:34:53 +0000

After a great first day, day three at Agile 2009 is a wrap. I have some general observations about the conference, but I’ll save those until next week to offer an overall recap. I missed out on most of Day 2 due to a pretty full meeting schedule. I was hoping Day 3 for me would begin with listening to Neal talk about Emergent Design and Evolutionary Architecture, but unfortunately I wasn’t able to make that session. I’ll have to track him down.

I started the day attending the third session of Programming with the Stars. Jeff and Joshua kickstarted today’s competition with a song that was written by Joshua titled, “50 Ways to Leave your Debugger”, which was a variation of Paul Simon’s song. Yes, they sang it. No, it wasn’t pretty. Sorry guys.

There were only three of the original six teams remaining, and each team was provided six minutes to complete their agile development task today (as compared to the three and a half minutes on day 1). It was pretty obvious that the pairs were starting to figure out the Stars competition, as they put on a much more polished performance today. For those who care about completely irrelevant statistics, there were 2 Macs and what appeared to be a Dell netbook left in the competition.

The programming topic today was story test driven development. For those not familiar with story TDD, it’s essentially starting off development with high level customer or acceptance tests instead of low level unit tests. Gerard and Ola put on a great performance and were the clear winners of the day, meaning they were granted immunity from getting cut. That brought it down to James and Kenrick versus J.B. and Llewellyn, and the initial crowd vote was close enough to require two separate recounts. Eventually James and Kenrick won out, pitting them against Gerard and Ola for the final day of competition. Some highlights of the competition include:

J.B.’s comment on their inability to finish the exercise in stating, “If we’d have done it in Ruby like we wanted, we would’ve gotten done.”

James great explanation of the home automation story he and Kenrick would be coding, along with very clear descriptions of exactly what they were doing at all times.

Gerard simulating a completely random experience when drawing their story from a stack of cards. Only later was it revealed that each card contained the same story.

The final day of competition promises to be an exciting event. Later this afternoon, I took in a session titled “Scrum and CMMI: From Good to Great - Are You Ready-Ready to be Done-Done.” I was expecting a bit more information on the relationship between Scrum and CMMI, but this session didn’t follow the title all that closely. The session thesis was pretty simple, though the numbers hard to believe, and the concept not crystal clear. By institutionalizing Scrum across all project teams, two teams within an organization were able to quadruple their productivity, and I discovered the intent of this session was to explain what they did to achieve such dramatic gains. That’s cooler than CMMI anyway, right?

Jeff Sutherland wasn’t listed as a speaker, but there were times when he hi-jacked the discussion and made some “salesy” claims about Scrum increasing productivity and decreasing time-to-market without offering a lot of insight as to how. From what I gathered though, they did this through two very simple concepts - by ensuring they were Ready to start a Sprint and were able to declare themselves Done after a Sprint. I never got a good sense of the attributes surrounding Ready, except that the team took great care in preparing the product backlog. Done was pretty easy to understand - basically all tests must pass. There were some very useful tidbits of information, which include the following:

A key element of Done is that testing was included in the definition. Testing commenced immediately after the code was written. Any delay between coding and testing only increased the cost of the effort.
It was imperative that the team keep the system functional, and minimize their technical debt. I’ve called this the prime directive of software development - always have a functional system. Because of this, the team had a fully integrated system at the end of every Sprint.
The team captured the important data that was necessary to provide them the information they needed to assess their improvement. This helped ensure they were moving in the right direction.

Along with a few great hallway conversations, a good third day of the conference. Looking forward to tomorrow.

@kirkk.com blog: Stars & Craftsmanship

Tue, 25 Aug 2009 02:53:01 +0000

The first day of Agile 2009 is in the books. While it’s great that the conference is packed with amazing sessions, it also means that it’s not possible to take it all in. Today’s highlights for myself included Programming with the Stars and the Software Craftsmanship tutorial led by UncleBob.

The Stars contestants did a great job and it was interesting to see the pairs work through some simple problems in such a short timeframe. It’s amazing what they were able to accomplish in only three and a half minutes. Languages represented in the contest included Java, Groovy and C. It’ll be a lot of fun to watch this play out through the remainder of the week.

Immediately after Stars, it was time to talk about the craft of software development. UncleBob is always an entertaining speaker who brings a lot of energy and passion to the stage. He began by highlighting his three laws of agile development (while also citing the sources from which he culled these nuggets of wisdom, which I have omitted here).

In preparing for battle, I have always found that plans are useless, but planning indispensable.
A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over, beginning with a working simple system.
…the document-driven, specify-then-build approach…lies at the heart of so many…software problems.

He then went on to talk about general best practices, including some pretty thought-provoking comments. He spoke of incremental improvement, never being blocked, avoiding big redesigns, going fast demands going well, and the importance of clean code. The discussion certainly resonated with me, and the irony in which he presented the material kept the atmosphere light. Two of his quotes caught my attention.

When talking about the importance of TDD versus fancy architecture and design documents, he stated:

If you have a great design, you’ll still be afraid to change the code. If you have a suite of tests, you won’t.

This isn’t to imply that design isn’t valuable, but that our emphasis on spending a lot of time designing the system might be a big misguided. Instead, the real measure of progress is working code and a rich suite of tests means we have the courage to ensure that code remains clean. Then, when talking about testing, he stated:

Manual tests are like me handing you the source code without a computer and telling you to execute it.

He acknowledged that some manual tests are important, but that most manual testing should be exploratory. Acceptance tests, unit tests, and many other forms of tests can, and definitely should, be automated. All in all, a great kickoff day for what stands to be a great week.

@kirkk.com blog: Modularity by Example

Thu, 13 Aug 2009 15:24:06 +0000

There are lots of benefits to modularity, some of which I discussed when introducing modularity patterns. But here’s a simple example, which serves as a prelude to some upcoming posts explaining a few of the patterns.

In the diagram at right (click to enlarge), the top left quadrant shows a sample system with a relatively complex class structure. When change occurs within a single class, shown in red in the bottom left quadrant, understanding the impact of change is difficult. It appears possible that it can propagate to any class dependent on the class highlighted in red. Assessing the impact of change requires that we analyze the complete class structure. The ripple effect appears significant, and change instills fear.

But if the system is modular with classes allocated to these modules, as shown in the bottom right quadrant, then understanding the impact of change can be isolated to a discrete set of modules. And this makes it much easier to identify which modules contain classes that might also change, as shown in the top right quadrant. Change is isolated to classes within modules that are dependent on the module containing the class that is changing.

This is a simple example, but it serves as evidence of the need for modular architecture, and illustrates one reason why modularity is so important. Modularity makes understanding the system easier. It makes maintaining the system easier. And it makes reusing system modules much more likely. As systems grow in size and complexity, it’s imperative that we design more modular software. That means we need a module system for the Java platform. It means that module system shouldn’t be shielded from enterprise developers. And it means we need to understand the patterns that are going to provide the guidance necessary in helping us design more modular software.

@kirkk.com blog: Agile Architecture at Upcoming Conferences

Tue, 11 Aug 2009 14:43:42 +0000

This fall, I’ll be speaking at a few conferences on agile architecture. Interestingly, a lot of the ideas expressed in these talks are not new to me. I gave a session titled From Code to Architecture at the now defunct SD Expo back in 2006 (give or take a year). Later that year, I gave a talk on Agile Architecture at SD Best Practices. Now, however, the technology is starting to catch up to the concepts I discuss in the presentations, making them easier to apply. The conferences I’m attending and the sessions I’m leading are listed below:

Agile RX/Ruby RX - A 90 minute session titled Agile Architecture. Again, present the basic mechanics of agile architecture, with an emphasis on technology. That means showing some code.
SpringOne 2GX - A 90 minute session titled Agile Architecture - Technologies and Patterns. Again, presents the basic mechanics of agile architecture, but this talk serves as the lead-in session to a collection of other sessions related to OSGi. The importance of modularity will obviously be discussed.
OOPSLA - A half-day tutorial titled Agile Architecture via Modularity Patterns. This session is going to include lots of coding exercises. We’ll examine the impact of modularity on architecture, and obviously we’ll talk about OSGi. We’ll walk through a simple application, and simulate typical architectural shifts that occur throughout the development lifecycle. It’s really quite amazing to witness the architectural transformation that takes place. This will be my first OOPSLA experience, and there are some pretty cool colocated conferences that I’m looking forward to.
Agile Development Practices - A 90 minute session titled Agile Architecture - Patterns and Technology. This session presents the basic mechanics of agile architecture, and we’ll take a look at some code. Also, colocated with this conference is the Agile Leadership Summit, which is pretty cool.

Session details vary slightly, but the general theme is consistent. To get a feel for the theme, take a look at a few of the blog entries where I’ve expressed my ideas surrounding agile architecture. If you happen to be attending any of these conferences, please feel free to seek me out. I’m always interested in a conversation on the topic. The blog entries are listed below.

Agile Architecture - My views on agile architecture and the natural architectural shifts that occur throughout the development lifecycle.
Agile Architecture Requires Modularity - Discusses the role of modularity in agile architecture.
On SOLID Principles and Modularity - Discusses where you need flexibility in architecture.
Two Faces of Modularity and OSGi - Introduces the need for patterns and tools to help design more flexible and modular architecture.
Reuse: Is the Dream Dead - Discusses the tension between reuse and use.
Modularity Patterns - Presents 19 modularity patterns that help ease the tension between reuse and use while making a software system easier to understand, maintain, and extend.