Is Special’s Stat Man Backing Off? – March 4

4 Comments

• 

  joedmello

  March 5, 2007 at 8:22 pm

  My Correspondence with Dr. Andrey Feuerverger
  Over the past four days, I corresponded with Dr. Andrey Feuerverger via e-mail and challenged the media interpretation of his 600:1 odds result. The Discovery Channel interpretation is (exact quote from http://dsc.discovery.com/convergence/tomb/about/about.html):

  “It comes down to a matter of statistics. A statistical study commissioned by the broadcasters (Discovery Channel/Vision Canada/C4 UK) concludes that the probability factor is 600 to 1 in favor of this tomb being the tomb of Jesus of Nazareth and his family.”

  In my email to Dr. Feuerverger, I claimed that the correct statistical interpretation is:

  Statistically Correct Claim: “It comes down to a matter of statistics. A statistical study commissioned by the broadcasters (Discovery Channel/Vision Canada/C4 UK) concludes that the probability factor is 600 to 1 in favor of this tomb being the tomb of Jesus of Nazareth and his family, ON THE CONDITION THAT THE JESUS (OF NAZARETH) FAMILY HAD A FAMILY TOMB TO BEGIN WITH.”

  As you can see, this makes a HUGE difference! If you factor in this difference and revise the calculations, the chances that the tomb in question belong to the NT Jesus family drops from a whopping 599/600 (a near certainty, per the shows claims) to probably no more than a measley 1 in 10 chance (or even less), which deflates the show’s probability claims!

  Yesterday, Dr. Feuerverger acknowledged that my revised interpretation was correct, and he was professional enough to make the corresponding changes to his web page – see email chain below (which the previous post appears to refer to, in part at least), but he said that it was not possible for him to get the media to make any corrections or retractions. However, he also seemed to have no objection to my trying to influence the media to do that. I am posting here on this blog in the hope that someone can take up this cause (or help me to do that) to ensure that the media interpretation of the 600:1 odds is corrected and the true statistical odds (for whatever they are worth!) are made known to the public.

  For those of you interested in the actual probability and statistics, you can see the articles referring to my challenge of the 600:1 odds claim on the New Testament Gateway blog (http://ntgateway.com/weblog/). The titles of the articles are ‘The “Jesus Family Tomb” Statistics: Further Developments’ (March 04) and “The correct interpretation of Dr. Andrey Feuerverger’s 1:600 odds calculation” (March 02).

  If you just want the summary of my discussion with Dr. Andrey Feuerverger, please read the following e-mail exchange with Andrey Feuerverger appended below (read it from the BOTTOM UP to get the right sequence).

  Date: Mon, 5 Mar 2007 00:18:54 -0800 (PST)
  From: Joe D’Mello
  Subject: OK, Thanks!
  To: Andrey Feuerverger

  Hi Andrey,

  OK, thanks! I’ll do what little I can to get the correct interpretation out to the media.

  Joe
  ======================================
  Andrey Feuerverger wrote:

  hi joe,

  any assistance you can render in this matter will, of course, be appreciated.
  it is simply not possible for me to entertain doing this just now.

  all best
  andrey f.

  ======================================
  Date: Sun, 4 Mar 2007 16:27:19 -0800 (PST)
  From: Joe D’Mello
  Subject: Re: Issue with media INTERPRETATION of your 600:1 odds calculations
  To: Andrey Feuerverger
  CC: [email protected], [email protected], [email protected]
  Hi Andrey,

  Thanks for your prompt response, and I am glad that you believe that I am “essentially correct” in my claim that the result of your 600:1 odds calculation is being misrepresented in the media. Indeed, it is impossible for you to correct every misquote that appears in the media, and I certainly appreciate that. However, would I be asking too much of you if I made the request that you kindly ensure that Discovery Channel – which commissioned your study – quote your findings correctly.

  Clearly, Discovery’s quote (which I reproduced in my e-mail from earlier today) does not align with the changes you have made to your website. Other media are merely quoting from the Discovery Channel, so why not at least correct the source? I believe that Discovery should correct the statement and publish (in SIMPLE LAYMEN’s ENGLISH, not statistical lingo) a retraction of their original interpretation of the 600:1 odds. This is just responsible and honest reporting, and the public is entitled to it!

  If you don’t have the time to do that, would you mind if I contact Discovery Channel and possibly other media (ABC, CBS, etc.) and state that you agree with the modified interpretation of the 600:1 odds number?

  Thanks again for your responsiveness!

  Best regards,

  Joe
  ======================================
  Andrey Feuerverger wrote:

  hi joe

  i appreciate your message. as you can well imagine i am now so deluged
  that i literally cannot correct every error or misquote that is out there.
  you may perhaps be able to do this for me, but understand if you cannot!
  in response to your query, you are essentially correct on two points,
  if i understand you correctly, and my web page has two changes in it
  which i attach to the end of this message so you don’t have to search
  needlessly.

  with best
  andrey feuerverger

  ————

  A `null hypothesis’ can be thought of here as asserting that
  this cluster of names arose purely by chance under random sampling
  from the onomasticon. The alternative hypothesis is the opposite
  of this, in some sense. It is not in the purview of statistics
  to conclude whether or not this tombsite is that of the New
  Testament family. Any such conclusion much more rightfully
  belongs to the purview of biblical historical scholars who are
  in a much better position to assess the assumptions entering
  into the computations. The role of statistics here is primarily
  to attempt to assess the odds of an equally (or more) `compelling’
  cluster of names arising purely by chance under certain random
  sampling assumptions and under certain historical assumptions.
  In this respect I now believe that I should not assert any
  conclusions connecting this tomb with any hypothetical one
  of the NT family. The interpretation of the computation should
  be that it is estimating the probability of there having been
  another family at the time whose tomb this might be, under certain
  specified assumptions.

  —————

  — We assume that this tombsite observation represents the `best’
  of many `trials’. It is estimated that there are approximately
  4000 inscribed male ossuaries and somewhat fewer than half as
  many inscribed female ossuaries in existence. The number of
  `trials’ is then taken as being approximately 1000.
  The computations do not take into account families who could
  not afford ossuary burials or who did not have sufficient
  literacy to have their ossuaries inscribed.

  ======================================
  Date: Sun, 4 Mar 2007 09:47:27 -0800 (PST)
  From: Joe D’Mello
  Subject: Issue with media INTERPRETATION of your 600:1 odds calculations
  To: Andrey Feuerverger
  CC: [email protected], [email protected], [email protected]

  Dear Professor Feuerverger,

  Thanks for your reply! I can certainly understand the need for a collective reply since you must be deluged with e-mail inquiries. I sincerely apologize for adding to your e-mail overload by replying to this message, but I feel professionally compelled to do so. I have read with interest your calculation and your recent explanation of it (in `The Tomb Computation’). Your explanation reinforces my understanding of your computation, but my issue is with the INTERPRETATION of your computation floating around in the media. For example, the Discovery Channel site makes the following claim (exact quote from http://dsc.discovery.com/convergence/tomb/about/about.html):

  Discovery Channel: “It comes down to a matter of statistics. A statistical study commissioned by the broadcasters (Discovery Channel/Vision Canada/C4 UK) concludes that the probability factor is 600 to 1 in favor of this tomb being the tomb of Jesus of Nazareth and his family.”

  After reading your computation and explanation, I remain convinced that your statistical computation supports the following claim:

  Statistically Correct Claim: “It comes down to a matter of statistics. A statistical study commissioned by the broadcasters (Discovery Channel/Vision Canada/C4 UK) concludes that the probability factor is 600 to 1 in favor of this tomb being the tomb of Jesus of Nazareth and his family, ON THE CONDITION THAT THE JESUS FAMILY HAD A FAMILY TOMB TO BEGIN WITH.”

  (In other words, you have computed a conditional probability.) It goes without saying that statistically, logically, and semantically the two claims are a world apart, and the first claim as stated by Discovery Channel is a subtle and shameful attempt to mislead the public. Your computation, as it appears to me, does not “adjust” for the whole Jewish populace that lived in the area in question during the entire time period in question, but only “adjusts” for the 1,000 tombs found in the area! Do you agree with the corrected claim above? If you do, I hope you will consider it your professional responsibility to ensure that your findings are correctly communicated by Discovery Channel.

  If you don’t agree with my corrected interpretation, please reply with a brief one-sentence email saying that you don’t. In that case, I will await your more detailed peer-reviewed paper (which you refer to in `The Tomb Computation’) and will cease to send you any further e-mails until then.

  By the way, I am copying Dr. Mark Goodacre, Associate Professor in New Testament at the Department of Religion, Duke University, North Carolina, on whose award-winning New Testament Gateway blog (http://ntgateway.com/weblog/) I have challenged the INTERPRETATION of your computation (see Friday, March 2, 2007 entry). If you could join us on that blog to discuss the interpretation of your computations by the media (or send Dr. Goodacre a response for him to post there), I would be truly delighted! I am also copying [email protected] (Ted Koppel’s e-mail question bucket) because I had sent my interpretation of your computations to him earlier this week.

  Best regards,

  Joe D’Mello
  ======================================

  Andrey Feuerverger wrote:

  Dear Statistical Colleague(s)

  I have been overwhelmed with E-mail messages regarding
  a certain calculation that has been widely reported
  in the media and am unable to send individual replies
  at the present time.

  Instead I have prepared a statement intended for the statistical
  community which hopefully addresses some of these questions.

  This information now appears on my Home Web Page; please hit
  the button marked `The Tomb Computation’.

  With best,
  Andrey Feuerverger

  ======================================
  Date: Wed, 28 Feb 2007 08:59:07 -0800 (PST)
  From: Joe D’Mello
  Subject: Fwd: Request for assumptions & calculations
  To: [email protected]
  CC: [email protected]
  Dear Professor Feuerverger,

  Since I did not hear back from you on the email I sent yesterday (copy below), I sent a formal request today to Discovery Channel requesting the assumptions and details underlying your calculations, and am also copying your president, Dr. David Naylor, on this email. I’m sure that as a respected faculty member of a university of worldwide repute, any professional assertions you make, especially in matters that have profound historical significance, will have sound documentation and analysis, and will pass the highest levels of academic scrutiny and peer review.

  The brief numerical calculation in the pdf document on The Discovery Channel website raises more questions than it answers, and it appears to me that the logic is flawed. However, I cannot be sure unless I can inspect the detail and assumptions underlying your calculations. Will it be possible for you to send me these? Better still, could you kindly post that detail (at a level comparable to that of a scholarly research publication) on the Discovery Channel website, so that fellow academics can have the opportunity to understand and appreciate your work?

  Best regards,

  Dr. Joe D’Mello

  ======================================
  Date: Tue, 27 Feb 2007 08:49:19 -0800 (PST)
  From: Joe D’Mello
  Subject: Request for assumptions & calculations
  To: [email protected]
  Dear Professor Feuerverger,

  As a fellow mathematician I am sending you this email to request your calculations (and associated assumptions) for the probability numbers being circulated in the media about the 600:1 odds (attributed to your calculations) that the tomb belonged to Jesus’s family. I am generally suspect of media coverage, and want to get the real scoop directly from you, so I can get a better understanding of the assumptions and the true interpretation of these odds. I would appreciate any information you can provide in this regard.

  Best regards,

  Joe D’Mello

  Chicago, USA
  • 

    bock

    March 5, 2007 at 9:49 pm

    New Stats facts!! – dlb
    Joe thanks for this. The exchange is interesting.
    • 

      joedmello

      March 6, 2007 at 6:31 am

      Using Feuerverger’s own calculation to deflate Discovery’s claim
      I had initially (last week) tended to ascribe the misinterpretation of the 600:1 odds calculation to Dr. Feuerverger. However, after carefully weighing Dr. Feuerverger’s words on the Sunday show and after our e-mail exchange (posted above), I feel reasonably certain that the massaging of the interpretation (as it appears on the Discovery webpage and as depicted on the show) is the handiwork of the show’s leadership, and not Dr. Feuerverger. On Ted Koppel’s panel discussion it became very clear (to me at least) that several experts were misquoted and that expert opinion was stretched, to say the least. I feel pretty certain that Dr. Feuerverger was also misquoted and the significance of his 600:1 odds distorted without this consent.

      Actually, I have no problem with Dr. Feuerverger’s computation if it is INTERPRETED CORRECTLY. He has made an honest attempt to use simple enumerative mathematics to perform a very basic calculation founded on the (perhaps flawed) assumptions given him. Yes, we can debate those assumptions and claim that different assumptions would yield different probabilities. Fine! But the truth of the matter is that if we interpret Dr. Feuerverger’s 600:1 odds correctly (as I suggested to him and he agreed – see my post above) we can get a far more relevant probability that the show’s leadership just can’t refute and which demolishes their statistical argument!

      In fact, I think this approach may be a better one initially because it does not challenge the 600:1 odds but clarifies and builds upon it to make a point that the show’s leadership will be compelled to accept – because their own statistician will support it! How can they suddenly disagree with their own statistician?

      In the email exchange above, Dr. Feuerverger agreed that the 600:1 odds are to be interpreted as follows:

      Suppose A is the event that the particular tomb in question is that of the NT Jesus family, and suppose B is the event that the NT Jesus family had a tomb (among the roughly 1,000 tombs found to date in the area). Then, Feuerverger’s 600:1 odds claim means (per his agreement) that

      P(A|B) = 599/600

      (This is to be read, “The probability of A, given that B has occurred is 599/600”, i.e., almost certain.) The symbol “P” represents the probability of the event following it in parentheses. A basic theorem in probability states that

      P(A and B) = P(A|B) x P(B) (x is multiplication)

      (Here, P(A and B) is the probability that BOTH A and B occurred.)

      Note that Simcha and the media are either confusing P(A and B) with P(A|B) or they are claiming “strategic ignorance” by choosing to not acknowledge the difference – which is common sense to mathematicians and statisticians.

      Now, let’s try to calculate P(B). I’ll make the following roughly right assumptions here based on my readings to date (correct me if you have more precise numbers):

      Assumption 1: There were roughly 150,000 Jews (men and women) who lived in the time period in question in the geographic area in question

      Assumption 2: The average Jewish family size at the time was 5

      Assumption 3: There are about 1,500 family tombs (I recollect that 1,000 were found, so let us assume that there are another 500 not found) belonging to the population referenced in Assumption 1 above.

      This means that there were roughly 150,000/5 = 30,000 Jewish families of whom only 1,500 had family tombs. So, a family PICKED AT RANDOM would have only a 1,500/30,000 = 1/20 chance of having a family tomb. Since Dr. Feuerverger has used “randomness” in all his calculations to date, we can use his very approach to claim then that the chance that the Jesus family had a family tomb to begin with is 1/20, and this is the P(B) we are after!

      Now we can get back to our theorem:

      P(A and B) = P(A|B) x P(B) = 599/600 x 1/20 = 599/12,000 = 4.99% or 5% approximately.

      This means that the probability that the NT Jesus family had a family tomb which is the one is question is approximately 5%. Or, there is only a 1 in 20 chance that this is the NT Jesus family tomb – per Dr. Feuerverger’s own 600:1 result and his own interpretation of it. How can Discovery refute it? If they do, they would be disagreeing with their own statistician!

      Of course, even if the three assumptions above need some adjustments, the conclusion won’t change significantly. We should not really quibble over whether there is a 1% or 2% or 5% or 15% chance because these numbers are ALL low enough to demolish the show’s claim. Let’s say the actual numbers in Assumptions 1, 2 and 3 above were off by a factor of two, and that we wound up with a 10% (1 in 10 chance) instead of 5% (1 in 20 chance) that the NT Jesus family had a family tomb which is the one is question. There’s no damage done because any number below 35% or so would be inconclusive anyway, since it would show that there could have been three or more families whose tomb was the one found, and that shatters the show’s claim.

      Your opinions?

      Joe
      • 

        bock

        March 7, 2007 at 5:07 am

        Using Feuverberger – dlb
        This looks quite good to me.