When does Evaluation Become Bureaucracy?

Submitted by Andrew Taylor on March 30, 2016 - 9:52am
Reflections on The Utopia of Rules by David Graeber
In my role as an evaluator, I try my best to create surveys and tools that are thoughtful, accessible, and useful.   Even so, I sometimes worry.  By the time my tools arrive in the hands of front line staff or volunteers who I have never met, do they understand the thinking behind the tool, or do they simply see yet another layer of boring paperwork?  Has my evaluation design morphed into bureaucracy? 

I have just finished reading The Utopia of RulesOn Technology, Stupidity, and the Secret Joys of Bureaucracy, by David Graeber.  It is a dense, difficult, sweeping, and sometimes brilliant book on the ways in which bureaucracy has permeated every aspect of our lives and constrained the way we think about democracy and about capitalism.   Although Graeber does not comment on program evaluation directly in his book, I suspect he might say that evaluation work is not only bureaucratic, but inherently and deliberately so.  And if evaluation is bureaucratic, then he would argue that it is oppressive.  His argument goes something like this. Bureaucracy is about trying to make the world fair and transparent by replacing human judgement and open ended social negotiation with a set of clearly explained and consistently enforced rules.  It is about trying to make society run like a formally organized sport. In sports, what counts as a goal or a touchdown is objective. What counts as a penalty is agreed-upon by everyone. How you succeed and how you fail is crystal clear. If you play by the rules and practice hard, you will win.  For the author, bureaucracy is fundamentally the idea that we can solve social problems and build a fair and just society if only we can create a set of rules and procedures that are up to the task. 

The problem, Graeber argues, is that this is a utopian vision. Like all utopian visions, it is something that can never be sustained in the real world.  Social problems are much more complex than the problems one faces in a hockey game. They can't be fixed by creating a better set of rules.  In any modern bureaucratic society, those who have power and privilege will always have the discretion to bend or break the rules.  Those who are treated unfairly will always seek to find loopholes within the rules, or opt out of participation in the game entirely.   The ultimate form of bureaucratic utopia, the author argues, is when we come to love the system of rules more than the people those rules are intended to help.  We buy into the fantasy that the rules are capable of creating a society that is fair and just.   If only, we say, you had filled in the form correctly, or read the small print, or completed your letter of intent  on time, the system would have worked properly.  It would have been fair and transparent, and you would have gotten the money, the support, or the consideration you deserved. 
Of course, no system of rules is ever perfect.  We have all seen corruption, selective use of the rules,  and lack of transparency in the system from time to time.   Paradoxically, we typically try to solve these problems by creating additional layers of bureaucracy.  We create boards of directors to oversee staff teams, and we create auditors and ombudspersons to monitor the boards.  Evaluators are as guilty of this mistake as anyone.   When evaluation projects fail to provide the answers we seek, we create more layers of evaluation.  Eventually, we create nested systems of rules so arcane that no-one really understands them. And then we blame the marginalized for failing to understand, thereby marginalizing them further.  In other words, Graeber concludes, we oppress.    

As evaluators, we might be seen as having contributed some uniquely perverse flourishes to our society's system of rules for addressing social issues.  This is especially true for evaluators like me who have worked with governments and other funders to design evaluation systems.  We have said that social interventions must be both innovative and evidence based. Not too many strategies can meet both those standards, and getting better at one of them often means you lose ground on the other.   Funders and evaluators have also said that programs must serve those with complex needs, but that the intended outcomes should be focused and measurable.   These stances could be seen as examples of rule sets that no nonprofit can fully obey, in a game that "the house" (in this case, whoever is funding and requiring evaluation) always wins. 

For Graeber, who is one of the key intellectual leaders of the Occupy movement, truly democratic interactions are those in which we set aside the rules and enter into open-ended, unstructured and non-hierarchical social negotiation.  In these interactions, we step together in the unknown.  We name and address power dynamics directly, and don’t pretend they can be legislated away using a system of rules.   

So - is evaluation inherently bureaucratic?  Is it necessarily and always an attempt to use surveys and forms as a way to avoid real, honest, open ended interaction with people?   I don’t think so. I don't believe I have fallen in love with my rules.  If you are engaging people in the process of design, and collecting data that you intend to use, and if you are asking questions that really matter both to you and to your participants and to other stakeholders, then the evaluation process won't become mindless paperwork.  Done properly, I've always thought evaluation can actually be a kind of social negotiation. It can be a way of starting conversations and building understanding across different groups of people - a process that can lead to real learning and action.  

Even so, I enjoyed reading this book.  It was a great reminder about how easily the good intentions of evaluators can degenerate into actions that are inadvertently bureaucratic and potentially oppressive.  Even if I don’t fall in love with the rules I create, it is possible that other people somewhere in the system will.  In my practice, I need to do all I can to prevent that from happening, by building evaluation designs that promote and reinforce social negotiation, rather than attempting to replace it.