The Vision of the Agile Data Method AgileData.org: Techniques for Disciplined Agile Database Development
Home  |  Agile DBAs  |  Developers  |  Enterprise Architects  |  Enterprise Administrators  |  Best Practices  |  Agility@Scale Blog  |  Announcements  |  Contact Us

Data is clearly an important aspect of software-based systems, a fact that the information technology (IT) industry has understood for decades, yet many organizations still struggle with their approach to data within their software processes.  As a consultant that has the privilege of working in a wide range of organizations, it seems that about one in ten organizations are reasonable successful with their approach to data-oriented activities, about six in ten think they’re doing well when they really aren’t, and the rest have a pretty good idea that they have a problem but don’t know what to do about it.  It doesn’t have to be this way.

So how do you know you’ve got a problem?  Enterprise data professionals, including both data architects and data administrators, will be frustrated by the fact that project developers on project teams ignore their advice, standards, guidelines, and enterprise models.  Worse yet the developers often don’t even know about these people and things in the first place.  Developers will be frustrated by what they perceive (often rightfully so) to be the glacial pace of enterprise data professionals to make or authorize seemingly simple changes. Database administrators (DBAs) often find themselves stuck in the middle of these two warring factions, trying to get their work done while struggling to keep the peace.  If one or more of these problems is common within your organization you’ve got a problem.

The goal of the Agile Data (AD) methodology is to define strategies that IT professionals can apply in a wide variety of situations to work together effectively on the data aspects of software systems.  This isn’t to say that AD is a “one size fits all” methodology.  Instead, consider AD as a collection of philosophies that will enable IT professionals within your organization to work together effectively when it comes to the data aspects of software-based systems.  Yes, this site also presents proven techniques that IT professionals can apply on the job, but the heart of AD is its underlying philosophies.  To understand why your organization should adopt the AD method you need to consider the following topics:

1. Why working together is currently hard

   • Warning signs that you've got a problem  

2. The agile movement

3. The philosophies of the Agile Data method

4. The Roles of the Agile Data method

5. Does Agile Data solve our problems?  

1. Why Working Together is Currently Hard

The relationship between data professionals and developers is often less than ideal within many organizations.  Yes, there are some organizations where these two communities work together quite well but there are always tensions – when a healthy tension exists between groups your organization can benefit, unfortunately these differences often lead to conflicts that aren’t healthy.  Your organization may not experience every single problem listed below although it likely suffers from a subset. The challenges that data professionals and developers must overcome can include:

1. Different visions and priorities.  Developers are often focused on the specific needs of a single project and often strive to work as much as possible in isolation from the rest of the organization.   DBAs focus on the database(s) that they are responsible for, often “protecting” the databases by minimizing changes to the them.  Data administrators and data architects focus on the overall data needs of the enterprise, sometimes to the virtual exclusion of the immediate needs of project teams.  Clearly the scope of each group is different, their priorities are different, and the issues that they deal with are different.  To make matters worse your project stakeholders, including direct users all the way up to senior management, have varying priorities and visions as well.
2. Over specialization of roles.  Specialists have a tendency to become too narrowly focused, to know everything there is to know about a small slice of software development but can become oblivious of everything else.  It is quite common to find senior Java developers that have never heard about data normalization, or even understand why you would want to do such a thing, and data architects that can’t read a Unified Modeling Language (UML) state chart diagram.  Because they are overly specialized they often have difficulties relating to other IT professionals. A common agile philosophy is that IT professionals should have a general understanding of the overall software process and have one or more specialties.  Because they are generalists they understand the broad range of issues pertinent to the “software game” yet at the same time have specific and valuable skills to offer to their team.  People who are just specialists or who are just generalists are at the extreme ends of the spectrum, as with most things in life it is far better to find a sweet spot in between these two extremes. What we really need are people who are generalizing specialists.
3. Process impedance mismatch.  One of the few things that processes such as the Unified Process, Extreme Programming, Scrum, DSDM, Crystal Clear, and Agile Modeling have in common is that they all work in an evolutionary (iterative and incremental) manner.  Unfortunately many within the data community still view software development as a serial or near-serial process. Clearly there is an impedance mismatch here, indicating that the data community needs to rethink their approach.  It is possible to take an evolutionary approach to data, a change that will require cultural and organizational changes within your organization.
4. Technology impedance mismatch.  Developers work with objects and components whereas data professionals work with databases and files.  Software engineering principles form the underlying foundational paradigm for objects and components whereas set theory forms the underlying foundational paradigm for relational databases (by far the most popular database technology).  Because the underlying paradigms are different the technologies don’t work together perfectly and an impedance mismatch exists.  This mismatch can be overcome although doing so requires a significant skillset.
5. Ossified management.  The technology and techniques used by IT professionals changes rapidly, a fact that we all know very well.  As people progress up the corporate hierarchy they deal less with technology and more with people issues, the end result being that managers have lost their technical edge.   The problem is that their previous development experiences, experiences on which they base technical decisions, may no longer be applicable.  We saw this as an industry when we moved from procedural technologies to object-oriented technologies – what may have been a good decision on a COBOL project often proves to be the kiss of death to a Java project.  We’re clearly seeing this problem once again as we move to agile software processes.  Management needs to change with the times.
6. Organizational challenges.  Common problems such as poor communications or politics between individuals and groups hurt the data aspects of software development just as badly as they hurt other efforts.
7. Poor documentation.  Most documentation seems to be at one extreme or another: little or no documentation or overly complex documentation that nobody reads.  Mutually agreed to development standards and guidelines, legacy system documentation, legacy database documentation, and enterprise models can be valuable resources when written well.  Agile documentation is definitely critical to your success.
8. Ineffective architectural efforts.  Most organizations face significant challenges when it comes to enterprise architecture, the most common challenge being that they don’t even know where to start.  Biased enterprise architectures, those that overly focus on one view of the enterprise, lead to architectures that do not adequately address the real needs of your organization.  As the Zachman Framework indicates, there are many potential views (which Zachman unfortunately refers to as components, a loaded term) that you want to consider, a concept captured by AM’s Multiple Models principle.  These views are data, function/process, network, people, time, motivation.   Ivory tower architectures, those formulated by teams that have removed themselves from the day-to-day realities of project teams, look good on paper but unfortunately fail in practice.  Developer’s unwillingness to conform to the constraints imposed upon them by enterprise architectural models, if they even know that such models exist, is also a common and serious problem.
9. Ineffective development guidelines.  Many organizations struggle to come to a collection of development guidelines that all IT professionals will work to.  There is a large number of causes for this, including people not understanding the need to follow such guidelines, people unwilling to follow someone else’s guidelines, overly complex guidelines, overly simplistic guidelines, a “one size fits all” attitude that leads to inappropriate guidelines for a specific platform, and an unwillingness to evolve guidelines over time.  When you have an effective collection of guidelines available to you, and everyone understands and applies them appropriately, you can dramatically improve the productivity of your IT efforts.  I maintain lists of coding guidelines and modeling style guidelines.
10.  Ineffective modeling efforts.  This is often the result of several of the previously identified problems.  People focused on a specific aspect of development will often produce models that wonderfully reflect the priorities of that narrow view but that don’t take into account the realities of other views.  An enterprise data model may present an excellent vision of the data required by an organization, but an enterprise model that reflects the data, functional, usage, and technical requirements of an organization is likely far more useful.  A UML class diagram may reflect the needs of a single project, but if it doesn’t reflect the realities of the legacy data sources that it will access then it is of little value in practice.  Modelers, and IT professionals in general, need to work together and look at the full picture to be truly effective.

1.1 Warning Signs That You've Got A Problem

It is very easy for organizations to deny that they have a problem.  It can be very difficult for senior management to detect problems until it’s too late because the “bad news” that they need to hear is filtered out long before it gets to them.  It can be difficult for people elsewhere in the organization to detect problems, perhaps everything is going quite well in their opinion – unfortunately the value system that they’re using to judge the situation isn’t ideal, making them blind to the problems that they are causing.  The following is a list of potential symptoms, each of which may indicate that your organization has one or more challenges that the Agile Data method may help you to address:

• People are significantly frustrated with the efforts, or lack thereof, of one or more groups.

• Software is not being developed, or if it is it is taking much too long.

• Finger pointing occurs, “the data administrators are holding up progress” or “the developers aren’t following corporate guidelines” are common complaints.  Worse yet, the finger pointer doesn’t perceive that they are also part of the problem.

• Political issues are given higher priority than working together to development, maintain, and support software-based systems.

• Ongoing feuds exist between people/groups.  Phrases like “you always” and “you never” are very good clues that feuds exist.

• Well-known problems within your organization are not being addressed.  Furthermore, suggestions for improvements appear to be ignored, nothing happens and no reason for rejection is provided.

• People are working excessively long hours with little or no reward.

• Decisions affecting teams, in particular project teams, are made in an apparently arbitrary and arrogant fashion.

We need to find a way to work together effectively.  There are clear differences between the data and development communities, and between the project and enterprise communities.  The fact that we’re talking about different communities is also part of the problem, arguably one of the roots causes.  You have a fundamental decision to make: Should you use these differences as an excuse to exacerbate existing problems within your organization or should you revel in these differences and find a way to take advantage of them?  I prefer the latter.  My experience is that the values and principles of the agile movement form the basis for an effective approach to working together.

2. The Agile Movement

To address the challenges faced by software developers an initial group of 17 methodologists formed the Agile Software Development Alliance (www.agilealliance.com), often referred to simply as the Agile Alliance, in February of 2001.  An interesting thing about this group is that they all came from different backgrounds, yet were able to come to an agreement on issues that methodologists typically don’t agree upon.  This group of people defined a manifesto for encouraging better ways of developing software, and then based on that manifesto formulated a collection of principles which defines the criteria for agile software development processes such as Agile Modeling.

The manifesto is defined by four simple value statements – the important thing to understand is that while you should value the concepts on the right hand side you should value the things on the left hand side (presented in red) even more.  A good way to think about the manifesto is that it defines preferences, not alternatives, encouraging a focus on certain areas but not eliminating others.  The Agile Alliance values:

1. Individuals and interactions over processes and tools. 

2. Working software over comprehensive documentation. 

3. Customer collaboration over contract negotiation. 

4. Responding to change over following a plan.   

The interesting thing about these value statements is there are something that almost everyone will instantly agree to, yet will rarely adhere to in practice.  Senior management will always claim that its employees are the most important aspect of your organization, yet insist they follow ISO-9000 compliant processes and treat their staff as replaceable assets.  Even worse, management often refuses to provide sufficient resources to comply to the processes that they insist project teams follow.  Everyone will readily agree that the creation of software is the fundamental goal of software development, yet insist on spending months producing documentation describing what the software is and how it is going to be built instead of simply rolling up their sleeves and building it.  You get the idea – people say one thing and do another.  This has to stop now.  Agile developers do what they say and say what they do. 

To help people to gain a better understanding of what agile software development is all about, the members of the Agile Alliance refined the philosophies captured in their manifesto into a collection of twelve principles.  Agile software development methodologies, such as Agile Modeling (AM) and the Agile Unified Process (AUP), should conform to.  These principles are:

1. Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

2. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

3. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter time scale.

4. Business people and developers must work together daily throughout the project.

5. Build projects around motivated individuals.  Give them the environment and support they need, and trust them to get the job done.

6. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

7. Working software is the primary measure of progress.

8. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

9. Continuous attention to technical excellence and good design enhances agility.

10. Simplicity – the art of maximizing the amount of work not done – is essential.

11. The best architectures, requirements, and designs emerge from self-organizing teams.

12. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

Stop for a moment and think about these principles.  Is this the way that your software projects actually work?  Is this the way that you think projects should work?  Re-read the principles once again.  Are they radical and impossible goals as some people would claim, are they meaningless motherhood and apple pie statements, or are they simply common sense?  My belief is that these principles form a foundation of common sense upon which you can base successful software development efforts.  A foundation that can be used to direct the data-oriented efforts of IT professionals.

For a detailed explanation of the values and principles of the agile movement, you may find Examining the Agile Manifesto of interest.

3. The Philosophies of Agile Data

The Agile Data method is defined by its six philosophies: Data. Enterprise issues.  Enterprise groups. Uniqueness. Teamwork. Sweet spot.

An interesting observation is that most of these philosophies aren’t specific to data, instead they are applicable to information technology efforts in general.  As the first principle implies you need to look at the overall picture and not just data, therefore data-specific principles very likely won’t serve you very well.  Heresy?  No, just common sense.

4. Roles with the Agile Data Method

The Agile Data method defines four roles – Application Developer, Agile DBA, Enterprise Administrator, and Enterprise Architect – that IT professionals will fulfill.  These roles, and how they work together, are discussed in greater detail in The Roles of Agile Data.  Table 1 summarizes the implications of the Agile Data method for IT professionals, implications that are also covered in The Roles of Agile Data.

Table 1. Implications for IT professionals.

5. Does Agile Data Solve Our Problems?

An important question to ask is whether the philosophies and suggested cultural changes address the problems that organizations face when it comes to the data aspects of software development.  Table 2 shows that this in fact is the case, listing the potential problems and the solution suggested by the Agile Data method.

Table 2. How the individual problems are addressed.

6. References

Agile Database Best Practices Agile/Evolutionary Data Modeling Agile Testing and Quality Strategies: Discipline Over Rhetoric The Criteria for Determining Whether a Team is Agile The Cultural Impedance Mismatch Between Data Professionals and Application Developers Evidence that Agile Software Development Scales The Process of Database Refactoring The Quest for the Silver Bullet Survey Results (Agile and Data Management) Why Agile Software Development Works: Improved Feedback This book describes the philosophies and skills required for developers and database administrators to work together effectively on project teams following evolutionary software processes such as Extreme Programming (XP), the Rational Unified Process (RUP), the Agile Unified Process (AUP), Feature Driven Development (FDD), Dynamic System Development Method (DSDM), or The Enterprise Unified Process (EUP).  In March 2004 it won a Jolt Productivity award. This book describes, in detail, how to refactor a database schema to improve its design. The first section of the book overviews the fundamentals evolutionary database techniques in general and of database refactoring in detail.  More importantly it presents strategies for implementing and deploying database refactorings, in the context of both "simple" single application databases and in "complex" multi-application databases.  The second section, the majority of the book, is a database refactoring reference catalog.  It describes over 60 database refactorings, presenting data models overviewing each refactoring and the code to implement it.   This book presents a full-lifecycle, agile model driven development (AMDD) approach to software development.  It is one of the few books which covers both object-oriented and data-oriented development in a comprehensive and coherent manner.  Techniques the book covers include Agile Modeling (AM), Full Lifecycle Object-Oriented Testing (FLOOT), over 30 modeling techniques, agile database techniques, refactoring, and test driven development (TDD).  If you want to gain the skills required to build mission-critical applications in an agile manner, this is the book for you.

Let Us Help

We actively work with clients around the world to improve their information technology (IT) practices, typically in the role of mentor/coach, team lead, or trainer.  A full description of what we do, and how to contact us, can be found at Scott W. Ambler + Associates.