Data Governance, as currently practiced, is failing. There have been some successes, but by and large, even these efforts have fallen short. Worse, many of those tasked with contributing to Data Governance find the effort painful. 

We have enormous sympathy for data governors. (We use the term “data governors” – DGs – as the most senior people charged with defining and implementing Data Governance.) Control is a fundamental management activity and so much in the data space is “out of control.” Further, data is increasingly important to all companies. The raw need for Data Governance is both enormous and growing.  

We Have Met the Enemy, and It Is Us!

What is going on? For the past few years we have dug deeper, researching why Data Governance fares so poorly:

In May 2020, we and six co-authors published the deliberately provocatively titled article “Data Management Has Failed! And Presented Us with a Historic Opportunity.” This article is also based on our continued research (sometimes including Doug Laney), Tom’s synthesis of why progress across the data space has been so slow, published in his new book, “People and Data,” and our ongoing work as advisors, mentors, and coaches.  

As expected, “Data Management Has Failed” generated both support and pushback. We found that some practitioners readily admitted the issues privately but were unwilling to do so publicly. Now we find that almost all agree privately and more and more are willing to discuss the matter publicly. We think this is a good sign for the profession.

We uncovered all the usual suspects: Lack of alignment, poor sponsorship, insufficient staff, poor change management, etc. However, none of these satisfied us, so we pushed harder, questioning everything. And it occurred to us: Maybe, just maybe, the problem is the DGs themselves! Part of the DG’s job is to create alignment, earn sponsorship, secure the needed budget, and drive change management (or get those qualified to do so on board). Perhaps DGs don’t have the full range of competencies they need, were misassigned or inadequately trained, or are philosophically unsuited for the job. Or perhaps they were tasked with the impossible, implicitly set up to fail. 

This led us to the work of Elliott Jaques. He advised that people had certain innate levels of ability along two (uncorrelated) dimensions [1]:

1. Their abilities to process increasingly complex and possibly disparate quantities of information
2. Their abilities to think into the future

To illustrate, a day trader may need to process lots of complex data, but they are only thinking a few minutes into the future, while an urban planner who also must process lots of complex data needs to think way into the future. See these points illustrated in figure 1. 

Now where do DGs fit? That question is made more complex because companies use the phrase Data Governance in many ways. To illustrate our thinking, consider this: The DG provides corporate-level oversight for the data program (e.g., is the data program designed in the best interest of the company and is it functioning as designed), on behalf of senior management and/or the board. It is a tough job, requiring both strategic thinking and considerable ability to think into the future, as depicted in figure 2.  

Also featured in figure 2 is the level of a hypothetical data governor on Jaques’ dimensions and a blue arrow indicating the gap between required and actual capabilities.

Note that one cannot fault the data governor for the gap which, according to Jaques’ theory, stems from innate abilities. This is key. An innate ability is inherent in an individual. For example, a new data governor may not have the connections that they need to be effective. But, making connections is a skill that can be learned, at least to some degree. Not so with the capability to think into the future. Human beings are, to some extent, wired differently.  

Now, you may be thinking, “That’s great, but not what we ask our data governor to do.” That’s too bad, because we believe that almost all organizations would benefit from this level of oversight.  

You can repeat the analysis we’ve illustrated above for your specific definition of Data Governance. Figure 3 locates the desired abilities of the data governor for two more situations.  

• Data Governance selects and operates a tool to provide a cataloging capability. Note that the hypothetical data governor is more than qualified for this job. 
• Data Governance defines and implements policies in the data space. Note that the hypothetical data governor has the innate ability to think far enough into the future but lacks the ability to process the required complex inputs.  

The issue we’re addressing here boils down to this: In the real world, are DGs innately capable of doing the jobs they are expected to do? Can they think far enough into the future? Can they handle the complexity of inputs needed to do the job?  

We fear that for many the answer to all three questions is a resounding “no.”

Our evidence that many of today’s data governors are not fully capable of doing their jobs, while anecdotal, comes from a variety of sources:  

1. A client recently stating, “I am having to do things I was not trained for.” This person did not see the challenges testing their innate capabilities and in learning new skills.
2. Another client said, “I don’t want to do the squishy stuff – our catalog and tools will take care of that.” We suspect this person cannot think into the future about how an organization will react to a new tool.
3. The chief data architect at a large organization rejected oversight over a major data strategy and architecture effort, stating, “I do not want anyone looking over my shoulder.” This person did not see that oversight would make acceptance of their models easier.
4. One person emphasized technology to close organizational gaps. While this could be a skills gap, we think it is more likely that the capability to see how technologies become embedded in organizations is missing.  

What to Do: Individual Data Governors

If, as a DG, all is going well, you need read no further. But if you are struggling, you and your boss should frame your analysis of what to do by first asking yourselves and your business counterparts the following question: Is it clear what is expected of the DG?

If not, the first order of business is to clarify expectations. This may take some thinking, as different parts of the business may have different expectations. Once the clear expectations hurdle is met, the next questions to ask are:

• What level of capabilities is needed to meet expectations, to do the job? What skills are also needed?
• What level of capabilities does the incumbent DG possess? What skills?
• Are there gaps and if so, how serious is it? Can they be closed?

You need to be brutally honest with yourself. We find that the best data architects can process incredible amounts of complex, often disparate information. But most struggle with the longer-term thinking needed to build the political support to define and implement significant new data policies. It is also hard to admit that, even though you are tasked to do a job that you have passionately seen as necessary for a long time, you might not have the capabilities to do the work. 

Note again that we want to draw a sharp distinction between innate capability and skill. Data Governance policies must be well-written, and the incumbent DG may be a poor writer. But this skills gap can be closed through training or use of a technical writer.  

Further, we sometimes see expectations of DGs that are simply unfair: For example, asking a highly talented data architect with no management experience to “spin up” a Data Governance office from scratch sets them up to fail. Yet, many new DGs, far removed from leadership, and with an initial Data Governance “team” of less than one FTE, are asked to do just that while also under incredible pressure. Hopefully, the root cause is just a profound lack of understanding on the part of leadership, not the use of a DG as a political chess piece.   

We understand one does not have to be qualified to do their job – being “the most qualified” is often good enough. In this case, transparency with leadership and business partners, remediating gaps, assignment of people who possess the missing skills, and a bit of patience from leadership may do just fine.

Fresh Thinking Needed 

Like any profession, too many data people cannot let go of established methods and techniques, such as “If we build it (e.g., a catalog, a policy, model, etc.), and toss it over to the business, things will be OK.” This inside-out thinking is past its sell-by date. 

It is time to think “outside in” and “out of the box.” It all begins and ends with customers; your business partners may not align with the classic cool stuff data professionals want to do. First, understand their needs, then put the right people in the right roles to meet them. As obvious as this statement may appear, it is far from standard practice. 

This may lead to some surprising results: For example, your DGs do not need to be data modelers, know how to write a policy, or explain a tech stack. These are skills that can be taught. Rather, consider DGs who know the business, can deal with complexity, have a track record with longer-term projects, and want to learn data.  

[1] Jaques, Elliott, Requisite Organization 2nd Edition, Routledge, 2016