Tuesday, February 22, 2011

Agenda Driven Data

About a year and a half ago, I was interviewed by a large Jewish organization regarding the Jewish Economics Survey I was conducting (and am looking to release a second version of shortly). Very quickly, the interview headed downward: The interviewer's questions were extremely leading, and he was clearly looking for me to answer in specific ways, primarily along the lines of "if someone would pump money into this, we'd be able to solve this problem" - or in other words, exactly the opposite of what I actually felt. When I would start to answer "well, not really...", I'd quickly get cut off and another question would be asked in its place. Unsurprisingly, the footage which included me was cut just before the video was aired at a large event the organization ran a few weeks later, which apparently championed the need for people to donate more money to help solve the Jewish world's economic problems.

Agenda driven data is one of the hardest obstacles to overcome - whether in the Orthodox world or the world as a whole. For all the obfuscation regarding the Iraq War, the most legitimate potential criticism is not whether the Bush administration had the data to justify invasion, but whether the gathering and weighing of that data was agenda driven in such a way that they were more prone to give it credence than they otherwise would or should have been. More recently, it was impressive to see three liberal publications discuss how liberal or other agenda driven bias is causing major issues in extremely important scientific fields by placing ideology over science.

In The New York Times about two weeks ago, an article about social psychologists' bias tells an eye-opening story:
[Dr. Jonathan Haidt] polled his audience at the San Antonio Convention Center, starting by asking how many considered themselves politically liberal. A sea of hands appeared, and Dr. Haidt estimated that liberals made up 80 percent of the 1,000 psychologists in the ballroom. When he asked for centrists and libertarians, he spotted fewer than three dozen hands. And then, when he asked for conservatives, he counted a grand total of three.
Prior to that, The Atlantic noted that the structure of academia caused research to emphasize the sensational over fact, stating in Lies, Damned Lies, and Medical Science that
To get funding and tenured positions, and often merely to stay afloat, researchers have to get their work published in well-regarded journals, where rejection rates can climb above 90 percent. Not surprisingly, the studies that tend to make the grade are those with eye-catching findings. [...] Imagine, though, that five different research teams test an interesting theory that’s making the rounds, and four of the groups correctly prove the idea false, while the one less cautious group incorrectly “proves” it true through some combination of error, fluke, and clever selection of data. Guess whose findings your doctor ends up reading about in the journal, and you end up hearing about on the evening news?
And The New Yorker had a fascinating piece entitled The decline effect and the scientific method about the same time discussing how studies which were (and are) used to explain and create numerous theories and ideas are increasingly difficult to replicate, from the effectiveness of anti-depressants to memory to numerous other fields, causing huge questions to be raised about the efficacy and accuracy not only of the studies but of all that has been based on the findings of them. No matter the study, the more it is replicated, the less true it seems to be.

If we as a Jewish community wish to begin fixing our problems, examples such as these show we can't use agenda driven studies to come to conclusions. If we do, ultimately we'll rush in one direction that looks good, only to find over and over that it's just not working the way it's supposed to - and then it will be too late. When the JES began two years ago, the primary purpose was to put together a simple guide for young singles and couples to help them when they began living independently or started their marriage. As the data came in, however, its findings taught so much about what problems existed and how people viewed those problems and the community's economics as a whole - and pointed to, but didn't prove, what might be able to help.

It is high time we put in the effort to collect and truly understand as much data as possible about the Jewish community and its various underlying problems and their causes - and its strengths and what allows those to thrive. Perhaps (!) a data-driven approach will allow us to ultimately help the Jewish community, rather than simply push off what seems to be an increasingly close, inevitable collapse of the structure currently in place.


  1. I've been following SerandEz for a long time but I can't recall: was there ever a post with the results from the survey?

  2. QED - I discussed various results, but I didn't publish everything. I talked to a lot of people and was using their advice to tweak for a better version. Perhaps I'll put up more info shortly regardless, though - I got a bit busy (as you'll see from posts coming soon) and it unfortunately had to take a back burner.

  3. What happens when the main beneficiaries of the status quo are also the only people with the spare cash to run a comprehensive survey?

  4. Anon - Hehe! Great question. Luckily, conducting a true comprehensive survey shouldn't cost too much; and surely those who are currently subsidizing everyone else would be interested. :)

  5. To paraphrase Dr. House, if religious people interpreted data without an agenda, there would be no religious people.

  6. This comment has been removed by the author.

  7. (I don't want you to think that's a cheap shot -- I think it's one of the biggest problems with encouraging religious beliefs that fly in the face of reality, such as several at the heart of Orthodox Judaism. No matter how well people compartmentalize -- only using empiricism and unflinching skepticism in this area of life, but not that one -- it's bound to spill over.)

  8. Ariella - Thanks.

    JA - I had a feeling there might be a comment along those lines. I believe we disagree on a number of related issues, not worth getting into on this post. I think anyone could agree to the point of this post particularly in regards to how Orthodoxy approaches education. (Or, say, how the US government approaches debt. :) )

    I don't believe that God or religion will be proven nor disproved via empirical data, and if I'm not mistaken you mostly agree with that. So for this subject at least I'd prefer to keep that separate.