 |
Agile/Lean Data Governance Best
Practices
AgileData.org: Techniques for Disciplined Agile
Database Development |
 |
|
|
 |
The goal of
data
governance is to ensure the quality, availability, integrity, security,
and usability within an organization. The way that you go about this
is up to you. Many traditional approaches to data governance seem to
struggle in practice, I suspect in part because of the
cultural impedance mismatch but also in part because traditional IT
governance struggles in general. The command and control approach typical of
traditional governance strategies is a lot like herding cats, you do a lot
of work but nothing much gets accomplished in the long run.
Agile/lean governance, on the other hand, is focused on enabling people and
motivating them to do the right things. |
|
The goal of agile/lean data governance is to enable
development teams to maintain and develop high-quality data assets within your
overall IT ecology. A lean data governance
approach promotes a healthy, collaborative
relationship between data professionals and the teams
that they're supporting. Recently Per Kroll and I have been working
on how to take a lean/agile approach to governance software development projects
which has resulted in an
IBM Whitepaper. The paper presents a collection of practices, many of which
are applicable to IT governance in general (including data governance). The
approach is based on the observation is that the most effective way to govern
the actions of intellectual workers is through motivating and enabling them, not
by a command-and-control process. This philosophy is backed up by findings from
Dr. Dobb's Journal's
Current State of Data Quality Techniques survey which found that a
collaborative approach data management is more effective than a
command-and-control approach, which in turn is better than no approach at all.
Unfortunately, traditional approaches to IT governance are often implemented in
a command-and-control fashion.
This article is organized into the following sections:
1. Potential Challenges with Traditional Data
Governance Strategies
Traditional data governance strategies often suffer from one or more common
problems:
- Data governance doesn't fit into the overall IT governance effort.
In a way it's sort of questionable to even consider data governance, or
development governance, or SOA governance, or a myriad of other governance
efforts, outside the scope of the overall IT governance strategy. You
need to optimize the whole governance strategy, not just individual parts.
- Data governance efforts are ignored. The
2006
DDJ survey into the current state of data management practices showed
that 66% of development teams will choose to "work around" their
organization's data group, see Figure 1 below.
This is clearly problematic and an indication that your data governance
efforts can't possibly succeed if you can't find ways to collaborate
effectively with development teams.
- Data governance is too onerous. As you see in
Figure 1 20% of development teams report that the data group within
their organization is too difficult to work with. In some
organizations this includes data governors.
- Data governors are too slow to respond.
Figure
1 shows that 36% of development teams find that their data groups work
too slowly, motivating developers to what they believe is best (even though
they might not actually know what the best course of action is).
- Data governors aren't perceived as providing value.
Figure 1 shows that 19% of development teams report
that they believe that their data group doesn't add much value, often
because of the additional bureaucracy involved with traditional approaches.
|
 |
Figure 1. Reasons why development teams go around data groups.
2. Agile/Lean Data Governance Practices
In addition to supporting
agile database best practices, the agile/lean development governance
practices which you should consider adopting for your data governance efforts
are:
- Valued Corporate Assets. Guidance
(such as database design conventions,
modeling style guidelines,
data naming conventions, and report design guidelines), metadata definitions, and reusable
assets such as frameworks and components, will be adopted if they are
perceived to add value to developers. You want to make it as easy as
possible for developers to comply to, and more importantly take advantage
of, your corporate IT infrastructure. When data standards are
sensible, easy to understand, and easy to access then there is a
significantly greater chance that people will actually follow the standards
in practice. When you force people to conform to standards, when it
make it onerous for them to do so, then
you reduce the chance that they will actually do so.
- Scenario-Driven
Development. The whole cannot
be defined without understanding the parts, and the parts cannot be defined
in detail without understanding the whole. By taking a scenario-driven, also
called a usage-driven approach, you can understand how people will actually
use your system, thereby enabling you to build something that meets their
actual needs. A common mistake made by traditional data approaches is
that they take data-driven approaches (understandable, given their biases)
which gets them in trouble because data is too narrowly focused to drive
things and it doesn't reflect the needs of your overall governance effort.
- Include data professionals as active participants on development teams.
When your DM group is external to project teams it can foster a “them vs.
us” mentality within your IT organization if you’re not very careful. You
don’t need to have an external group to run your data governance activities,
instead individual data professionals can do so as part of their
responsibilities on development teams in a collaborative and timely manner.
This is one of the fundamental concepts of the
Agile Data method.
- Educate developers. Developers need to understand why your MDM efforts
are important, what the benefits are, and how to work together with your DM
team. When they know why something needs to be done, and how to do it
effectively, chances are much better that they’ll actually do it.
- Adapt the Process. Because teams vary
in size, distribution, purpose, criticality, need for oversight, and member
skillset, one process size does not fit all. The implication is that your
approach to support data-oriented activities, including governance, will
vary by team.
- Align Team Structure
With Architecture. The organization of your data team should reflect the
enterprise architectural structure. For example, a centralized data
team will struggle to support an environment with a decentralized
architecture.
- Align HR Policies With
IT Values. You need to establish specific incentives/rewards
appropriate for the mindset of your technical staff to ensure that they
follow your data governance strategy.
- Align Stakeholder
Policies and IT Values. Your development efforts are driven and
constrained by your stakeholders. Your stakeholders must be realistic in
their demands on IT, and understand the implications of their decisions
(you’ll need to educate them).
- Business-Driven Project
Pipeline. You should invest in the IT activities that are well-aligned
to the business direction, return definable value, and match well with the
priorities of the enterprise. This includes data-oriented activities as
well. Unfortunately, many traditional data governance strategies often
seem to reflect the priorities of the data bureaucrats among us, not the
priorities of the business, resulting in data warehouses etc. being built
which are underused
- Embedded Compliance.
It is better to build compliance into your day-to-day processes instead of
having a separate compliance process which often results in unnecessary
overhead. Automation is critical. For example, Instead of holding reviews
to ensure that development teams follow corporate data conventions,
a time consuming and expensive endeavor, why not instead run a static
code analysis tool against the database schemas on a regular basis to ensure
that the data naming conventions are followed?
- Flexible Architectures.
Architectures that are service-oriented, component-based, or object-oriented
and implement common architectural and design patterns lend themselves to
greater levels of consistency, reuse, and adaptability.
- Pragmatic Governance
Body. Effective governance bodies focus on enabling development teams in
a cost-effective and timely manner. They typically have a small core staff
with a majority of members being representatives from the governed
organizations.
- Promote Self-Organizing
Teams. The best people for planning work are the ones who are going to
do it. IT professionals should be respected as intelligent people who can
determine their own strategies for working together. When given a bit of
coaching and guidance, they can plan their work within established
parameters, such as iteration boundaries. Self-organization doesn't
mean that the team is out of control, any given team must conform to your
overall governance strategy, enterprise architecture, and so on.
- Risk-Based Milestones.
You want to mitigate the risks of your project, in particular business and
technical risks, early in the lifecycle. You do this by having throughout
your project several milestones that teams work toward. The goal of each
milestone is to address one or more risks, such as the
Lifecycle Architecture (LCA) milestone in the
Rational Unified Process (RUP) that requires your architecture be proven
through working code before Construction begins, thereby addressing
technical risk.
I've found that the following factors are critical to the success of your
data governance efforts:
- Recognize that IT governance is the real goal. Data
governance is just one part of your IT governance program, and it's highly
coupled to other aspects such as development governance, security
governance, and so on. Focusing just on data governance puts you at
risk of optimizing data governance in such a way that it doesn't work well
with the rest of your governance efforts, putting the entire governance
program at risk. Remember the first
agile data philosophy that data is only part of the overall picture.
- The governance effort must be owned. If someone is not
responsible for the IT governance effort it will very likely die a quick
death in your organization. My experience is that the people most
suited to be the owners of IT governance are the executive business
stakeholders. When it comes to data governance, the people least
suited to govern are data management professionals as they're the ones being
governed (amongst others). In short, don't let your data governance
efforts devolve into yet another political ploy of your data management
group to try to grab additional power.
- Have clear, quantifiable goals. What are you trying to
achieve? Improved quality? Improved productivity? Improved
time to value? Improved stakeholder satisfaction? Combinations
thereof?
- Measure and honestly report the results. It's easy to talk about
quantifiable goals, but it takes a fair bit of integrity to live up to your
promises (and I'll let the very serious lack of data around the value of
data governance efforts speak for itself). It's fairly easy to manage
direct costs such as the salaries of the people involved with the governance
effort, but a bit more difficult to measure indirect costs such as the
opportunity cost of potentially lengthening decision and development life
cycles due to increased governance (this issue is particularly acute for
traditional governance strategies). Measuring the benefits can also be
challenging, although as Doug Hubbard points out in
How to Measure Anything: Finding the Value of Intangibles in Business it
is possible if you think outside the box a bit. Automating metrics
collection is an important aspect of lean governance.
- Less is more. You need a lot less governance than what the
pro-governance people believe, although probably a bit more governance than
what the anti-governance people think. If you find that you need more
governance it's a lot easier to add it than it is to remove unnecessary
governance activities once they're in place.
- Educate the people affected. If the people involved,
including those being governed, don't understand what you're trying to
achieve and also believe that any additional effort on their part is
worthwhile then your governance effort will quickly fall apart.
Furthermore, this sort of education is ongoing.
4. Suggested Readings
 |
This book describes the philosophies and skills required for
developers and database administrators to work together effectively on
project teams following evolutionary software processes such as Extreme
Programming (XP), the
Rational Unified Process (RUP), the
Agile Unified
Process (AUP), Feature Driven
Development (FDD), Dynamic System Development
Method (DSDM), or The Enterprise Unified Process (EUP). In March 2004
it won a Jolt Productivity award. |
 |
This book describes, in detail, how to
refactor a database schema
to improve its design. The first section of the book overviews the fundamentals evolutionary database techniques in
general and of database refactoring in detail. More importantly it
presents strategies for implementing and deploying database refactorings, in
the context of both "simple" single application databases and in "complex"
multi-application databases. The second section, the majority of the
book, is a
database refactoring reference catalog. It describes over 60 database refactorings, presenting
data models overviewing each refactoring and the code to implement it.
|
 |
This book presents a full-lifecycle,
agile model driven
development (AMDD) approach to software development. It is one of the
few books which covers both object-oriented and data-oriented development in
a comprehensive and coherent manner. Techniques the book covers
include Agile Modeling (AM),
Full Lifecycle Object-Oriented Testing (FLOOT),
over 30 modeling techniques,
agile database techniques,
refactoring, and
test driven development (TDD). If you want to gain the skills required to
build mission-critical applications in an agile manner, this is the book for
you. |
|
|
We actively work with clients around the world to
improve their information technology (IT) practices,
typically in the role of mentor/coach, team lead, or trainer. A full
description of what we do, and how to contact us, can be
found at Scott W.
Ambler + Associates.
