Data Architecture? What's That?Data Architecture? What's That?

Data architecture has always been a topic close to my heart. "Overloaded and overlooked" is how I define a data architect's role. Has anything changed over the past decade? Yes, but possibly for the worse.

Years ago, I wrote an article in Intelligent Enterprise on the dot-com data architect. It was very well-received and also was the beginning of a long relationship with IE. Some DAMA (Data Management International) chapters invited me to speak on the topic. I accepted the offer from the Washington DC chapter (less travel) and was presenting to the audience near Union Station in DC... even as an airplane loaded with innocent passengers overhead was being coerced into a steep dive into the Pentagon. Then, everything changed.Has anything changed in data architecture since then? Consider.

Back then:

So, what has changed?

Well, for one, data architecture (and the data architect's job) has just gotten much harder. After the dot-com bust -- as organizations began to more maturely reap the benefits of distributed computing, SOA architectures and ubiquitous web applications -- data exploded in terms of size, diversity and complexity.

Then, over the past few years, we witnessed the coming of age of data quality awareness, master data management, data warehousing, business intelligence, business rules management, BPM and compliance reporting. More data sources and consumers, more data integration, more complexity, more stakeholder awareness... and more difficulties in data governance and management.

And meanwhile, what about the unassuming data model? Object and XML data models -- after making some threatening noises -- found and settled down in enterprise niches. Dimensional models, having first garnered attention by screaming out how they were NOT like relational models, have tamed down after getting sufficient business traction and realized that they aren't quite so different from relational models after all. Animated discussions on the "best" approach to data warehousing (that usually generated more heat than light) have largely cooled off. Master data management solutions -- a remarkable modern ode to creative technology marketing -- find that they are more ubiquitous in discussions and presentations than in terms of actual implementation. Developers are being told that they don't need data modelers at all; they can now generate data models right through their procedural code.

All along, in a competitive frenzy as yet unabated, companies are acquiring, merging, retiring and relegating data management technologies at an unprecedented pace.

And in the midst of this maelstrom stands the beleaguered data architect, alone and unappreciated. Even as millions of dollars are burnt up all around him (or her) in repeated tragicomic parodies of project management and solution architectures, the data architect -- the unseen force holding together the protons and neutrons of data in the nucleus of IT solutions -- must struggle to justify her (or his) existence.

Phooey, you say; passion for a cause does not justify it. But what other option is there? The Forresters and Gartners of this world do not, as far as I know, conduct any studies on the need or efficacy of data architecture and data architects. See? Ignored again!

This being a blog, I have a limited number of words to convey my message. So in conclusion, repeat after me: "Data architecture is a crucial piece of the software, and an experienced data architect is a critical success factor for any software project." If you don't have one in-house, run to the nearest consulting shop and get one. You won't regret it -- and your predecessor is already wishing she had one.

You're welcome.Data architecture has always been a topic close to my heart. "Overloaded and overlooked" is how I define a data architect's role. Has anything changed over the past decade? Yes, but possibly for the worse. Consider the following...