Data is clearly an important aspect of software-based systems, a
fact that the information technology (IT) industry has understood for decades,
yet many organizations still struggle with their approach to data within their
software processes. As a consultant
that has the privilege of working in a wide range of organizations, it seems
that about one in ten organizations are reasonable successful with their
approach to data-oriented activities, about six in ten think they’re doing
well when they really aren’t, and the rest have a pretty good idea that they
have a problem but don’t know what to do about it. It
doesn't have to be this way.
So how do you know you've got a problem? Enterprise data professionals, including both data
and data administrators, will be frustrated by the fact that project developers
on project teams ignore their advice, standards, guidelines, and enterprise
models. Worse yet the developers
often don’t even know about these people and things in the first place.
Developers will be frustrated by what they perceive (often rightfully so)
to be the glacial pace of enterprise data professionals to make or authorize
seemingly simple changes.
Database administrators (DBAs) often find themselves
stuck in the middle of these two warring factions, trying to get their work done
while struggling to keep the peace. If
one or more of these problems is common within your organization you’ve got a
The goal of the Agile Data (AD) methodology is to define
strategies that IT professionals can apply in a wide variety of situations to
work together effectively on the data aspects of software systems.
This isn’t to say that AD is a “one size fits all” methodology.
Instead, consider AD as a collection of philosophies that will enable IT
professionals within your organization to work together effectively when it
comes to the data aspects of software-based systems.
Yes, this site also presents proven techniques that IT professionals can
apply on the job, but the heart of AD is its underlying philosophies.
To understand why your organization should adopt the AD method you need
to consider the following topics:
working together is currently hard
philosophies of the Agile Data method
The Roles of the Agile
Agile Data solve our problems?
The relationship between data professionals and developers is often less than
ideal within many organizations. Yes,
there are some organizations where these two communities work together quite
well but there are always tensions – when a healthy tension exists between
groups your organization can benefit, unfortunately these differences often lead
to conflicts that aren’t healthy. Your
organization may not experience every single problem listed below although it
likely suffers from a subset. The challenges that data professionals and
developers must overcome can include:
- Different visions and priorities.
Developers are often focused on the specific needs of a single
project and often strive to work as much as possible in isolation from the
rest of the organization. DBAs
focus on the database(s) that they are responsible for, often
“protecting” the databases by minimizing changes to the them.
Data administrators and data architects focus on the overall data
needs of the enterprise, sometimes to the virtual exclusion of the immediate
needs of project teams. Clearly the scope of each group is different, their
priorities are different, and the issues that they deal with are different.
To make matters worse your project stakeholders, including direct
users all the way up to senior management, have varying priorities and
visions as well.
- Over specialization of roles.
Specialists have a tendency to become too narrowly focused, to know
everything there is to know about a small slice of software development but
can become oblivious of everything else.
It is quite common to find senior Java developers that have never
heard about data normalization, or even understand why you would want to do
such a thing, and data architects that can’t read a Unified Modeling
Language (UML) state chart diagram. Because
they are overly specialized they often have difficulties relating to other
IT professionals. A common agile philosophy
is that IT professionals should have a general understanding of the overall
software process and have one or more specialties.
Because they are generalists they understand the broad range of
issues pertinent to the “software game” yet at the same time have
specific and valuable skills to offer to their team.
People who are just specialists or who are just generalists are at
the extreme ends of the spectrum, as with most things in life it is far
better to find a sweet spot in between these two extremes. What
we really need are people who are
- Process impedance mismatch.
One of the few things that processes such as
Disciplined Agile Delivery (DAD),
Scrum, DSDM, Crystal Clear, and
Agile Modeling have in common is that they all work in an
and incremental) manner. Unfortunately
many within the data community still view software development as a serial
or near-serial process. Clearly there is an impedance mismatch here,
indicating that the data community needs to rethink their approach.
It is possible to take an evolutionary approach to data, a change that will require
organizational changes within your organization.
- Technology impedance mismatch.
Developers work with objects and components whereas data
professionals work with databases and files.
Software engineering principles form the underlying foundational
paradigm for objects and components whereas set theory forms the underlying
foundational paradigm for relational databases (by far the most popular
database technology). Because
the underlying paradigms are different the technologies don’t work
together perfectly and an impedance mismatch exists.
This mismatch can be overcome although doing so requires a
- Ossified management. The
technology and techniques used by IT professionals changes rapidly, a fact
that we all know very well. As
people progress up the corporate hierarchy they deal less with technology
and more with people issues, the end result being that managers have lost
their technical edge. The
problem is that their previous development experiences, experiences on which
they base technical decisions, may no longer be applicable.
We saw this as an industry when we moved from procedural technologies
to object-oriented technologies – what may have been a good decision on a
COBOL project often proves to be the kiss of death to a Java project.
We’re clearly seeing this problem once again as we move to agile
software processes. Management
needs to change with the times.
- Organizational challenges. Common
problems such as
poor communications or politics between individuals and
groups hurt the data aspects of software development just as badly as they
hurt other efforts.
- Poor documentation.
Most documentation seems to be at one extreme or another: little or
no documentation or overly complex documentation that nobody reads.
Mutually agreed to development standards and guidelines, legacy
system documentation, legacy database documentation, and enterprise models
can be valuable resources when written well. Agile
documentation is definitely critical to your success.
- Ineffective architectural efforts. Most organizations face significant challenges when it comes
to enterprise architecture, the most common challenge being that they
don’t even know where to start. Biased
enterprise architectures, those that overly focus on one view of the
enterprise, lead to architectures that do not adequately address the real
needs of your organization. As
the Zachman Framework indicates,
there are many potential views (which Zachman
unfortunately refers to as components, a loaded term) that you want to
consider, a concept captured by
Multiple Models principle.
These views are data, function/process, network, people, time,
Ivory tower architectures, those formulated by teams that have
removed themselves from the day-to-day realities of project teams, look good
on paper but unfortunately fail in practice. Developer’s unwillingness to conform to the
constraints imposed upon them by
enterprise architectural models, if they
even know that such models exist, is also a common and serious problem.
- Ineffective development guidelines.
Many organizations struggle to come to a collection of development
guidelines that all IT professionals will work to.
There is a large number of causes for this, including people not
understanding the need to follow such guidelines, people unwilling to follow
someone else’s guidelines, overly complex guidelines, overly simplistic
guidelines, a “one size fits all” attitude that leads to inappropriate
guidelines for a specific platform, and an unwillingness to evolve
guidelines over time. When you
have an effective collection of guidelines available to you, and everyone
understands and applies them appropriately, you can dramatically improve the
productivity of your IT efforts. I maintain lists of
guidelines and modeling
- Ineffective modeling efforts. This is
often the result of several of the previously identified problems.
People focused on a specific aspect of development will often produce
models that wonderfully reflect the priorities of that narrow view but that
don’t take into account the realities of other views.
An enterprise data model may present an excellent vision of the data
required by an organization, but an enterprise model that reflects the data,
functional, usage, and technical requirements of an organization is likely
far more useful. A
diagram may reflect the needs of a single project, but if it doesn’t
reflect the realities of the legacy data sources that it will access then it
is of little value in practice. Modelers,
and IT professionals in general, need to work together and look at the full
picture to be truly effective.
The data community appears to
have spent it's intellectual capital during the 1970s and 1980s.
Sadly, they seem to have gained little benefit from their
1.1 Warning Signs That You've Got A Problem
It is very easy for organizations to deny that they have a
problem. It can be very difficult
for senior management to detect problems until it’s too late because the
“bad news” that they need to hear is filtered out long before it gets to
them. It can be difficult for
people elsewhere in the organization to detect problems, perhaps everything is
going quite well in their opinion – unfortunately the value system that
they’re using to judge the situation isn’t ideal, making them blind to the
problems that they are causing. The
following is a list of potential symptoms, each of which may indicate that your
organization has one or more challenges that the Agile Data method may help you
We need to find a way to work together effectively.
There are clear differences between the data and development communities,
and between the project and enterprise communities.
The fact that we’re talking about different communities is also part of
the problem, arguably one of the roots causes.
You have a fundamental decision to make: Should you use these differences
as an excuse to exacerbate existing problems within your organization or should
you revel in these differences and find a way to take advantage of them?
I prefer the latter. My
experience is that the values and principles of the agile movement form the
basis for an effective approach to working together.
People are significantly frustrated with the efforts,
or lack thereof, of one or more groups.
Software is not being developed, or if it is it is
taking much too long.
Finger pointing occurs, “the data administrators are
holding up progress” or “the developers aren’t following corporate
guidelines” are common complaints. Worse
yet, the finger pointer doesn’t perceive that they are also part of the
Political issues are given higher priority than working
together to development, maintain, and support software-based systems.
Ongoing feuds exist between people/groups.
Phrases like “you always” and “you never” are very good clues
that feuds exist.
Well-known problems within your organization are not
being addressed. Furthermore,
suggestions for improvements appear to be ignored, nothing happens and no
reason for rejection is provided.
People are working excessively long hours with little
or no reward.
Decisions affecting teams, in particular project teams,
are made in an apparently arbitrary and arrogant fashion.
To address the challenges faced by software developers an initial
group of 17 methodologists formed the Agile Software Development Alliance (www.agilealliance.com),
often referred to simply as the Agile Alliance, in February of 2001.
An interesting thing about this group is that they all came from
different backgrounds, yet were able to come to an agreement on issues that
methodologists typically don’t agree upon.
This group of people defined a manifesto for encouraging better ways of
developing software, and then based on that manifesto formulated a collection of
principles which defines the criteria for agile software development processes
such as Agile Modeling.
The manifesto is defined by four simple
value statements – the important thing to understand is that while you should
value the concepts on the right hand side you should value the things on the
left hand side (presented in bold) even more. A good way to think about the manifesto is that it defines
preferences, not alternatives, encouraging a focus on certain areas but not
eliminating others. The Agile
The interesting thing about these value statements is there are
something that almost everyone will instantly agree to, yet will rarely adhere
to in practice. Senior management
will always claim that its employees are the most important aspect of your
organization, yet insist they follow ISO-9000 compliant processes and treat
their staff as replaceable assets. Even
worse, management often refuses to provide sufficient resources to comply to the
processes that they insist project teams follow. Everyone will readily agree that the creation of software is
the fundamental goal of software development, yet insist on spending months
producing documentation describing what the software is and how it is going to
be built instead of simply rolling up their sleeves and building it.
You get the idea – people say one thing and do another.
This has to stop now. Agile
developers do what they say and say what they do.
To help people to gain a better understanding of what agile
software development is all about, the members of the Agile Alliance refined the
philosophies captured in their manifesto into a collection of
twelve principles. Agile
software development methodologies, such as
Scrum and more importantly Disciplined Agile Delivery (DAD), should conform
to. These principles are:
Individuals and interactions over processes
Working software over comprehensive documentation.
Customer collaboration over contract negotiation.
Responding to change over following a plan.
Stop for a moment and think about these principles.
Is this the way that your software projects actually work?
Is this the way that you think projects should work?
Re-read the principles once again. Are
they radical and impossible goals as some people would claim, are they
meaningless motherhood and apple pie statements, or are they simply common
sense? My belief is that these
principles form a foundation of common sense upon which you can base successful
software development efforts. A
foundation that can be used to direct the data-oriented efforts of IT
For a detailed explanation of the values and principles of the
agile movement, you may find
Examining the Agile
Manifesto of interest.
An interesting observation is that most of these philosophies
aren’t specific to data, instead they are applicable to information technology
efforts in general. As the first
principle implies you need to look at the overall picture and not just data,
therefore data-specific principles very likely won’t serve you very well.
Heresy? No, just common
Our highest priority is to satisfy the customer through early and
continuous delivery of valuable software.
Welcome changing requirements, even late in development. Agile processes
harness change for the customer's competitive advantage.
Deliver working software frequently, from a couple of weeks to a couple
of months, with a preference to the shorter time scale.
Business people and developers must work together daily throughout the
Build projects around motivated individuals.
Give them the environment and support they need, and trust them to get
the job done.
The most efficient and effective method of conveying information to and
within a development team is face-to-face conversation.
Working software is the primary measure of progress.
Agile processes promote sustainable development. The sponsors,
developers, and users should be able to maintain a constant pace indefinitely.
Continuous attention to technical excellence and good design enhances
Simplicity – the art of maximizing the amount of work not done – is
The best architectures, requirements, and designs emerge from
At regular intervals, the team reflects on how to become more effective,
then tunes and adjusts its behavior accordingly.
The Agile Data method defines four roles – Application Developer,
Agile DBA, Enterprise Administrator, and
– that IT professionals will fulfill. These
roles, and how they work together, are discussed in greater detail in The
Roles of Agile Data. Table
1 summarizes the implications of the Agile Data method for IT professionals,
implications that are also covered in The
Roles of Agile Data.
Table 1. Implications for IT
An important question to ask is whether the philosophies and
suggested cultural changes address the problems that organizations face when it
comes to the data aspects of software development. Table 2 shows that this in fact is the
case, listing the potential problems and the solution suggested by the Agile
Table 2. How the individual problems are
- Everyone must agree to a common vision as to what they are trying to
accomplish and how they’re going to accomplish it.
- Agile software developers are willing to work with others and
co-locate as needed.
- Documentation is a reality of software development.
Choose to be agile in your approach.
- Agile software developers strive to be generalists with one or more
- Agile software developers are flexible in their approach because one
“process size” does not fit all.
- Software is your primary goal, enabling the next effort is your
- Application developers must work closely with project stakeholders.
- Application developers must recognize that legacy systems and
databases place constraints on them.
- Application developers should follow their organization’s
standards and guidelines and be willing to provide feedback into their
- Application developers will work closely with enterprise architects,
people who should become active members of a project team.
- Agile DBAs work very closely with application developers to
implement and support data-oriented development efforts.
- Agile DBAs must be willing to work in an iterative and
incremental manner, just as application developers do.
- Agile DBAs will work with enterprise administrators to take
advantage of and to help evolve corporate meta data, standards, and
- There is more to enterprise administration than data administration.
- The primary goal of enterprise administrators is to support project
team efforts, helping to guide the teams towards solutions that
reflect the overall needs of the enterprise.
- Enterprise administrators develop and support collections of
flexible standards and guidelines that reflect the actual needs of
developers, not the artificial needs of bureaucrats.
- Enterprise administrators support and work with others in the
organization to communicate the constraints imposed by the current
- Enterprise architects focus on more than just data architecture.
- Enterprise architects work in an iterative and incremental manner.
- Enterprise architects actively work with project teams to
communicate the architecture, to mentor project teams in architecture
and modeling skills, and to gain real-world feedback that they use to
evolve the architecture.
Different visions and priorities
Agile Data implores IT professionals to work together, to
understand and respect the viewpoints of your co-workers.
Over specialization of roles
Agile Data asks IT professionals to find the “sweet
spot” between the extremes of being a generalist and being a specialist
by becoming someone who is a generalist with one or more specialties.
Process impedance mismatch
Agile Data implores enterprise and data professionals to be
prepared to work following an incremental and iterative approach, the norm
for most modern development and the defacto standard for agile software
development. It also implores application developers to recognize
that the existing environment, and future vision for the organization,
places constraints on their efforts.
Technology impedance mismatch
Agile Data requires that IT professionals work together
closely, learning from each other as they do so. Agile DBAs
have the skills to map the application schema to the data schema, to write
data-oriented code, and to performance tune their work.
Agile Data asks enterprise architects to work with senior
management and educate them in the realities of modern software
development. Similarly application developers should work with and
help to educate both line and middle management.
Agile Data implores IT professionals to work with one
another and with your project stakeholders, to respect them and to
actively strive to work together effectively.
Agile Data directs IT professionals to follow the principles
Ineffective architectural efforts
Agile Data advises enterprise architects to take a
multi-view/model approach to architecture and to actively work on project
team to support and prove their architecture. The feedback from
these efforts should then be reflected in future iterations of the
Ineffective development guidelines
Agile Data implores enterprise administrators to write
clear, effective, and applicable standards and guidelines and to be
prepared to act on feedback from the development teams.
Ineffective modeling efforts
Agile Data directs IT professionals to follow the principles
and practices of the Agile Modeling methodology.