Adopting Evolutionary/Agile Database Techniques

www.agiledata.org: Techniques for Successful Evolutionary/Agile Database Development

This site has presents a wide range of philosophies, proven techniques, and one or two reasonable theories about effective ways to go about the data-oriented aspects of software development. What it hasn’t done is discussed what you need to do to succeed at them. Until now. in this article I describe how your organization can adopt the agile database techniques described at this site. My expectation is that it will be difficult for many organizations to adopt agile database techniques. It’s not because there is any inherently complex about them, the problem is due in the most part to cultural inertia.

To successfully adopt these philosophies and techniques you must:

Change the way you look at software development
Understand the challenges you face
Actually try it
Block non-agile co-workers
Be realistic
Let us help you become agile

1. Change the Way You Look at Software Development

The philosophies and techniques described at this site have several significant implications for your organization. My experience is that you must accept that:

Everyone needs to work closely together. Software development is a communication game, and as Cockburn (2002) argues, documentation is the worst form of communication available to you whereas face-to-face communication standing around a whiteboard is the best. Simple things such as co-locating the team in a single workspace, using simple tools such as whiteboards and paper, and having project stakeholders as active members of your team will improve your productivity by at least an order of magnitude.
The models and documents are finished when the software ships. It’s an iterative and incremental world now, not a serial one. The requirements are fully defined for the release of a system, if at all, only until the software is ready to go to final acceptance testing. Until then they might change. The same thing can be said of your logical data model, if you even create one, and your physical data model. These work products evolve as work progresses on the system and may even change just hours before the production baseline of your system is finalized.
Agile requires significantly greater discipline. Although this isn't what the traditionalists want to hear, the reality is that Agile approaches require significant greater discipline on the part of practitioners compared with traditional approaches.
Power within your IT department will shift. Changes in process always entail shifts within the power structure of an organization, and Agile Data is no exception. Agile database techniques shift the way that enterprise concerns, particularly those pertaining to data, are considered. Architectural model reviews are a thing of the past because your models evolve over time and because reviews are a “process smell” in the agile development indicating that you’ve made an organizational mistake earlier in your project. One way to look at it is if someone is qualified to review a work product why didn’t you involve them in its initial development? Existing organizational structures, particularly those based on specialized skills or a command and control structure, need to be reworked in favor of a more organic structure.
Everyone needs to become actively involved. The day of the specialist who attends reviews, or has work products submitted to them for review and feedback, is over. The day of the bureaucrat who merely pushes paper and has no direct influence on the creation, maintenance, operation, or support of a system should never have come about in the first place. Bureaucrats can’t hide behind their onerous and prescriptive processes anymore, instead they must roll up their sleeves, work closely with project teams, and actually add value to your IT efforts. People who fight this concept are likely good candidates for re-education and in extreme cases may need to be made aware of opportunities in other organizations (if you get my meaning). We need generalizing specialists now.
Everyone needs to rethink their approach and beliefs. There are many thought provoking ideas presented at this site. It is possible to take an evolutionary approach to database design. There is more to modeling than the UML, or data for that matter. There are many different ways that you can approach development, one size does not fit all. Many technical issues often thought to be the sole domain of databases, including both referential integrity and transaction control, are also pertinent within your objects. You have many technical options available to you, and they all have their strengths and weaknesses. The point is that these ideas are likely to go against what you currently believe, or how you currently prefer to work. Get over it.

2. Understand the Challenges you Face

You also need to understand the challenges that you are likely to face when introducing agile data techniques into your organization. My experience is that the most difficult challenges that you will need to overcome are not technical in nature but instead are people oriented. At the risk of promoting stereotypes, I have found that experienced IT professionals, novice developers, and managers all seem to have their own unique challenges that need to be overcome. Let’s look at each group one at a time.

Experienced IT professionals, often people with twenty or more years in the industry, likely:

Haven’t invested the time to understand agile software development.
Haven’t taken, or been allowed, the time to try the techniques and philosophies described at this site.
Aren’t comfortable with change, particularly change that dramatically shifts the political power structure within your IT department.
Are convinced that their existing approaches work (which they do in some situations) so don’t see the need to change their ways.
Have had bad experiences in the past with “code and fix” (CAF) approaches. Because they don’t understand agile software development they often equate it with CAF and therefore assume that agile techniques are a bad idea. The article How Agile Are You? describes how to determine if a team is agile or not.
Have some very good points that aren’t well addressed by the agile community, such as data-oriented and enterprise issues, and therefore they feel that agile techniques aren’t sufficient for their needs.
Believe that their situation is unique, perhaps they work in a Fortune 50 company (although 49 other organizations are also in this situation) or a government agency, and therefore agile techniques won’t work for them.
Focus on symptoms, and not the root causes of problems within their IT organization, and therefore they haven’t questioned their preferred approach to development.
Haven’t coded in years and don’t realize the implications of the new techniques and tools currently being used by developers. Java/C# development is significantly different than COBOL development.
Have listened myths and misconceptions promoted by to other experienced IT professionals whom they respect, unfortunately people who are also struggling with agile concepts and whom are likely to tell them what they want to hear, and as a result feel that they don’t need to continue looking into agility.
Are narrowly focused specialists, often with years if not decades of experience, and therefore have difficulty understanding the bigger development picture.
Are likely scared that they don’t fit into agile development (and very likely don’t, given their current skillset). My June 2006 column in Dr Dobb's Journal addresses the issue of how traditionalists can become productive members of an agile team.

Novice IT professionals are also struggling with learning agile techniques, although they face different challenges. They likely:

Perceive agility as meaning they don’t have to model and they don’t have to write documentation.
Have very narrow experience, if any, as a programmer.
Don’t have the experience to appreciate the bigger picture, or at least to appreciate its nuances.
Are focused on a single technology or programming language.

Managers within your organization have their own unique issues regarding agility. They often:

Can’t provide adequate resources for your team (or simply don't understand what they are).
Have had bad experiences in the past with new techniques and is unwilling to try yet another one.
Believe that agile software development is another fad and will go away after a few years.
Don’t realize that an agile development team requires other parts of the organization, including both IT groups such as enterprise architects and data management/administration, to work in an agile manner as well when they interact with the team.

Everyone needs to approach agility with an open mind, ideally without any preconceptions. They need to look at the big picture, recognize that they have serious problems that aren’t going to go away unless they act. Ideally everyone needs to work with, and be mentored by, experienced agile developers in order to learn these new approaches. Although it is possible to bootstrap your project team, and even your entire organization, into agile software development you are much better advised to seek the help of people who have gone before you. Education and training in agile software development is also important although not as critical as good mentoring.

Experienced developers and managers need the opportunity to try these new ways, and more importantly be given the time it takes to break themselves of their “bad” non-agile habits. Novice developers need to focus on education/mentoring as well as simply gaining experience. Novice developers will likely learn agile techniques easier than experienced developers will as they have less baggage to discard along the way.

3. Actually Try It

There is a significant difference between theory and practice. You can read about something all you want, but until you try something you truly won’t understand it and its implications. At some point you’re going to have to get your feet wet and try this stuff out on a real project.

Remember the adage
"In theory, theory and practice are the same,
but in practice, they're not."

4. Block Non-Agile Co-workers

A common strategy is to start with a pilot project that tries the new techniques in practice so you can gain some insight into how they’ll work within your organization. Although this enables the individual team to be agile the challenge is that the rest of your organization is still following your existing, non-agile approach. The implication is that they may expect certain models or deliverables to be delivered at specific times in specific formats, or they may require status reports or other management artifacts, or they may require you to follow their non-agile procedures. Ideally you should negotiate away the need to venture into this bureaucratic morass, realistically you often can’t and therefore you put your pilot project at risk of failure (exactly what many of the bureaucrats are hoping for).

One way to overcome this problem is to assign one or two people on your team to be a blocker. In North American football the primary goal of a blocker is to prevent the other team from sacking your quarterback or tackling your receivers, either of which would cause your play to fail. In software development a blocker attends the meetings of the bureaucrats and produces the work products that they require, freeing up the rest of the team to focus on the activities that actually lead to building your system. The blockers effectively implement a “process façade” around your team that makes it appear to the rest of the organization that your team is following their existing procedures. This satisfies the bureaucrats, yet prevents them from meddling with the people that are doing the real work. Although it sounds like a wasted overhead, and it is because it would be far more effective to divert both the blockers and bureaucrats to efforts that produce something of value, the advantage is that it enables the rest of the team to get the job done. The role of blocker is often taken on by your team’s project manager or coach, although in the past I have let this be a revolving role on the project so as to spread out the pain of dealing with the paper pushers.

5. Be Realistic

The following list describes several important factors that you need to consider when bringing agile database techniques into your organization.

Be patient, it will take a generation. I believe that the adoption curve for agile software development, including agile database techniques, will be similar to that of object technology. Object technology was first being considered within the business community in the late 1980s and at the time of this writing many organizations, often referred to as the Late Majority or Laggards on the technology adoption curve (Moore 2002), are still struggling with adopting object orientation (OO). That’s fifteen years by my watch, and I expect no different for agile software development (the implication is that it may be 2020 before your organization finally catches up).
Don’t be too fervent. The surest way to turn someone off of agile techniques is to try to convince them that agile is the only way. Remember the sixth philosophy of Agile Data – recognize that you should strive to find the appropriate sweet spot between extremes. The implication is that you might not become as agile as you’d like right away, but you very like could become more agile than you currently are.
Don’t go in blind. Reading this book is a good first step towards becoming more agile in your data-oriented activities, but it’s only the first step. Do some more reading before you try to adopt the techniques and philosophies described at this site, www.agilealliance.com and www.agilemodeling.com are great resources that you should take advantage of. Talk with people who are following agile techniques on their own projects to discover what they’ve experienced. Get involved with the agile data community, the mailing list page lists good online resources, and share your own experiences.
Don’t underestimate the politics. Process is politics, and when you change the process as radically as I describe at this site
many people will obviously take issue with the necessary changes.
Be prepared to find work elsewhere. Your organization may not be ready for agility, and worse yet may not be for some time. The implication is that you have a very difficult decision to make – do you continue to wait and hope that an opportunity arises within your organization, do you try to create such an opportunity, or do you update your resume and start looking for work elsewhere. As Ron Jeffries likes to say “Change your organization or change your organization.” It’s your life, take control of it.

6. References and Suggested Online Readings

This book describes the philosophies and skills required for developers and database administrators to work together effectively on project teams following evolutionary software processes such as Extreme Programming (XP), the Rational Unified Process (RUP), the Agile Unified Process (AUP), Feature Driven Development (FDD), Dynamic System Development Method (DSDM), or The Enterprise Unified Process (EUP). In March 2004 it won a Jolt Productivity award.

This book describes, in detail, how to refactor a database schema to improve its design. The first section of the book overviews the fundamentals evolutionary database techniques in general and of database refactoring in detail. More importantly it presents strategies for implementing and deploying database refactorings, in the context of both "simple" single application databases and in "complex" multi-application databases. The second section, the majority of the book, is a database refactoring reference catalog. It describes over 60 database refactorings, presenting data models overviewing each refactoring and the code to implement it.

This book presents a full-lifecycle, agile model driven development (AMDD) approach to software development. It is one of the few books which covers both object-oriented and data-oriented development in a comprehensive and coherent manner. Techniques the book covers include Agile Modeling (AM), Full Lifecycle Object-Oriented Testing (FLOOT), over 30 modeling techniques, agile database techniques, refactoring, and test driven development (TDD). If you want to gain the skills required to build mission-critical applications in an agile manner, this is the book for you.

7. Let Me Help

I actively work with clients around the world to improve their information technology (IT) practices as both a mentor/coach and trainer. A full description of what I do, and how to contact me, can be found here.