Evolutionary/Agile Database Best Practices

www.agiledata.org: Techniques for Successful Evolutionary/Agile Database Development

Modern software development processes, including Rational Unified Process (RUP), Extreme Programming (XP), Agile Unified Process (AUP), and Scrum are all evolutionary if not agile in nature. The implication is that if data professionals are going to be effective members of such teams, then they need to adopt tools and techniques which enable them to do so. There is nothing special about the data aspects of an IT system: they can be developed in an evolutionary manner just like non-data aspects. Even data warehouse and business intelligence projects can be developed in an evolutionary manner (and quite frankly this is the preferred approach). This article overviews a collection of best practices for agile/evolutionary database development.

These "best practices" for evolutionary/agile database development are:

Database refactoring. A database refactoring is a small change to your database schema which improves its design without changing its semantics (e.g. you don't add anything nor do you break anything). The process of database refactoring is the evolutionary improvement of your database schema so as to improve your ability to support the new needs of your customers.
Agile data modeling. With an evolutionary approach to data modeling you model the data aspects of a system iteratively and incrementally. With an agile approach you do so in a highly collaborative manner. You will envision the initial requirements and architecture early in the project at a high-level and then model storm the details on a just in time (JIT) basis when and if they're needed -- no more "big up front modeling (BMUF)".
Database regression testing. You should ensure that your database schema actually meets the requirements for it, and the best way to do that is via testing. With a test driven development (TDD) approach you write a unit test before you write production database schema code, the end result being that you have a 100% regression test for your database schema. Agile testing provides the concrete feedback that you need to ensure data quality.
Configuration management of database artifacts. Your data models, database tests, test data, and so on are important project artifacts which should be configuration managed just like any other artifact.
Developer sandboxes. Developers need their own working environments, called sandboxes, where they can modify the portion of the system which they are building and get it working before they integrate their work with that of their teammates.
Data normalization. Data normalization is a process in which data attributes within a data model are organized to increase the cohesion of entity types. In other words, the goal of data normalization is to reduce and even eliminate data redundancy, an important consideration for application developers because it is incredibly difficult to stores objects in a relational database that maintains the same information in several places.
Set a realistic primary key strategy. The fact is that sometimes it makes sense to use natural keys and sometimes surrogate keys. As a professional you need to understand when to apply each strategy, and to be prepared to refactor if you discover that you've made the wrong choice.
Database encapsulation. A database encapsulation layer hides the implementation details of your database(s), including their physical schemas, from your business code. In effect it provides your business objects with persistence services – the ability to read data from, write data to, and delete data from – data sources. Ideally your business objects should know nothing about how they are persisted, it just happens. Database encapsulation layers aren’t magic and they aren’t academic theories; database encapsulation layers are commonly used practice by both large and small applications as well as in both simple and complex applications. Database encapsulation layers are an important technique that every agile software developer should be aware of and be prepared to use.
Train developers in basic data skills. This enables developers to both improve their data-oriented work and to interact with data professionals more effectively. Fundamental skills include: relational database fundamentals, data modeling, mapping objects to RDBs (O/R mapping), working with legacy data, XML, referential integrity and shared business logic, how to retrieve objects from an RDB, how to implement reports, security access control, and transactions and concurrency control.
Train DBAs in basic development skills. Similarly, DBAs need to gain an understanding of application development so that they can play active roles on the team(s) which they support.
Common development guidelines. Having a common, usable set of development standards which are easy to understand and to comply to can greatly improve the quality of the systems that you develop. These guidelines may include, but not be limited to, programming guidelines, modeling style guidelines, data naming conventions, and user interface conventions (including report design conventions).
Lean data governance. The goal of data governance is to ensure the quality, availability, integrity, security, and usability within an organization, and the goal of agile/lean data governance is to enable development teams to do these things effectively within your overall IT ecology. Many traditional approaches to data governance seem to struggle in practice, I suspect in part because of the cultural impedance mismatch but also in part because traditional IT governance struggles in general. The command and control approach typical of traditional governance strategies is a lot like herding cats, you do a lot of work but nothing much gets accomplished in the long run. Lean governance, on the other hand, is focused on enabling people and motivating them to do the right things. A lean data governance approach promotes a healthy, collaborative relationship between data professionals and the teams that they're supporting.
Agile master data management (MDM). If you're going to adopt an MDM strategy within your organization, it should at least be an agile one. Many organizations struggle when it comes to MDM, typically because they adopt a traditional, command-and-control strategy. Your MDM efforts can in fact be very agile and streamlined if you choose to.

References and Suggested Online Readings

This book describes the philosophies and skills required for developers and database administrators to work together effectively on project teams following evolutionary software processes such as Extreme Programming (XP), the Rational Unified Process (RUP), the Agile Unified Process (AUP), Feature Driven Development (FDD), Dynamic System Development Method (DSDM), or The Enterprise Unified Process (EUP). In March 2004 it won a Jolt Productivity award.

This book describes, in detail, how to refactor a database schema to improve its design. The first section of the book overviews the fundamentals evolutionary database techniques in general and of database refactoring in detail. More importantly it presents strategies for implementing and deploying database refactorings, in the context of both "simple" single application databases and in "complex" multi-application databases. The second section, the majority of the book, is a database refactoring reference catalog. It describes over 60 database refactorings, presenting data models overviewing each refactoring and the code to implement it.

This book presents a full-lifecycle, agile model driven development (AMDD) approach to software development. It is one of the few books which covers both object-oriented and data-oriented development in a comprehensive and coherent manner. Techniques the book covers include Agile Modeling (AM), Full Lifecycle Object-Oriented Testing (FLOOT), over 30 modeling techniques, agile database techniques, refactoring, and test driven development (TDD). If you want to gain the skills required to build mission-critical applications in an agile manner, this is the book for you.

Let Me Help

I actively work with clients around the world to improve their information technology (IT) practices as both a mentor/coach and trainer. A full description of what I do, and how to contact me, can be found here.