The Shakespeare Conference: SHK 18.0492 Wednesday, 1 August 2007
From: Ward Ward <
This e-mail address is being protected from spambots. You need JavaScript enabled to view it
>
Date: Wednesday, 01 Aug 2007 00:24:25 -0700
Subject: 18.0473 Shakespeare Golden Ear Test
Comment: RE: SHK 18.0473 Shakespeare Golden Ear Test
Our warmest thanks to the 80 brave SHAKSPERians who took our Golden Ear
Test, Round 1, and to Hardy for making them all reachable by us. We
hope that SHAKSPERians will indulge us in an oversize 20-page posting,
giving a preliminary analysis of the results of the test, which test ran
for ten days, from July 6 through July 15. It's now OK to discuss the
test online, as well as off, but try not to give away too many of the
test's specifics, to preserve some usability for it in the future. The
test is still up, http://goldenear.cmc.edu, but we haven't been
monitoring it since July 15 and would suppose that any results since
then are likely to be retakes to get another look at the test.
Our two-sentence conclusion is this: As individuals, on average, the
whole SHAKSPER group got almost two out of three unrecognized passages
right (63%), and the top 30% got almost three out of four right (74%).
As an aggregated group, the whole group got almost four out of five
right (79%), and the top 30% got almost five out of six right (82%).
For a two-paragraph conclusion, scroll down to Table II, and then to
Section XI, Conclusions, and don't miss the cautions and further thanks
at the very end.
For 20 pages of further detail, read on.
I. Who took the test?
31 of the 80 takers considered themselves Shakespeare pros (39%) and 49
considered themselves amateurs. We would guess that almost all were
from SHAKSPER, though there was also an admixture of takers from HLAS, a
rowdy, free-wheeling sister group of Shakespeare authorship buffs, most
of whom also belong to SHAKSPER but are less likely to be pros than
SHAKSPERians and, on their own turf, are subject to none of Hardy's
constraints. It's possible that a few of these do not belong to
SHAKSPER, but we don't think it matters much. HLAS people are scarcely
less hooked on Shakespeare, and, hence, almost as high on our list of
people who might take a test like ours seriously and give us interesting
results.
26 respondents described themselves as critics, 14 as writers, 33 as
artists, including performing artists, and 8 as "other humanities [than
literature], or social, or natural sciences." 80 is about 6% of
SHAKSPER's membership, a respectable showing. To encourage wider
participation, we did not require test takers to give their names, and
most did not, but 14 takers (18%) did identify themselves as willing,
and, in some cases, eager, to take our Round 2 test. 12 of these were
Rated, that is, they scored Bronze or better on the test. We did not
encounter many, if any, A-List Shakespeare celebrities, no Harold
Blooms or Stephen Greenblatts among the self-identifiers, nor many, if
any, of SHAKSPER's most vocal past advocates of shutting down your
computers and listening to your intuition only -- but we can't exclude
the possibility that some Shakespeare grandees or intuitionists might
have taken the test anonymously. Some A-list people helped us design
the test; we would not expect any of these to have taken it.
II. Gross Accuracy as Found: Two out of three for the group, four out of
five for the top 30%
Of the 80 who took the test, 24 (30%) were rated Bronze or better, with
the following distribution:
Table I. Rated Takers by Category, All Ratings Gross
Golden Silver Bronze Total
Identified 4 4 4
12
Unidentified 3 2 7
12
Sh. Pro 6 1 5
12
Sh. Amateur 1 5 6
12
Total 6 11 24
Rated players got four out of five identifications right, in gross
figures, three out of four right in net figures, and did better on
non-Shakespeare than on Shakespeare. Gross figures count all correct
answers, whether recognized from memory or detected by intuition; net
figures subtract the recognized answers and count intuitive answers
only. Net figures are the interesting ones, but they are harder to
arrive at than gross and are not available for all purposes.
Of the rated takers, neatly, half considered themselves Shakespeare
pros, the other half amateurs. Half identified themselves, half did not.
All of the ratings are slightly inflated because they are based on
gross, not net accuracy. Subtracting recognized passages would reduce
most Golden Ears to Silver, and most Silver to Bronze.
The whole group, on average, got about two out of three identifications
right, both in gross figures and in net, since the whole group
recognized fewer passages, on average, than the Rated players. Like the
Rated players, the whole group did better on non-Shakespeare than on
Shakespeare. The average gross score of all 80 takers was 18.6 of 28
(66%); their net score equivalent would be about a point lower, 17.6
(63%).
The 31 pros in the whole group scored between 14 and 25, averaging 18.9
right of 28 questions (68%, gross). The 49 amateurs scored between 14
and 24, averaging 18.4 right (66%, gross). It is not surprising that
the pros did better, on average, than the amateurs. It is surprising
that the gap is so small, especially considering that these are gross
scores uncorrected for passages recognized by the test-taker, which one
would expect to be more common among pros than among amateurs. In fact,
the average gross accuracy scores of every subgroup - critics, writers,
artists, others -- we tested fell into an extremely narrow range, none
lower than 18, none as high as 19 (Table X, below).
Should the two-out-of-three or three-out-of-four individual gross
accuracy levels we found be considered high or low? Judging from
post-test lamentations we have heard, scattered mutterings about
"humbling experience" and "stupid test," and low self-identification
rates even of high-scoring players, we would guess that many, perhaps
most, takers were disappointed with their scores, expected to do better,
and didn't want to put their names to their test results.
Our further guess is that they reacted to our test much as law students
were expected to do to an experiment/demonstration routinely performed
in Evidence classes in our day, decades ago. A two-minute dramatic
event is staged, and the students are asked to describe what happened.
Who was chasing whom? What did they say? What color were the first
one's eyes? How tall was the second one, and what did he weigh? What
color were his socks? The answers, when collected, always turn out to
be all over the lot and filled with inaccuracies, and the class is
supposed to learn from the "humbling experience" that eyewitness
testimony is not very reliable.
We followed a different model in our Golden Ear experiment (below). Our
expectations of individual performances were not so high, and we believe
that SHAKSPERians have done quite a bit better as a group than most of
them think. The average individual outcomes as found struck us as par
for the course or better; and those for rated players seem particularly
impressive. Moreover, we now have further confirmation that the scores
as found can be tweaked by screening and aggregation to reach
surprisingly high levels of group accuracy, which have never, to our
knowledge, been demonstrated before on this scale.
Let's start with the outcomes as found. Two-out-of-three average
accuracy, is far from perfect, but it is better than chance, better on
non-Shakespeare than most of our individual computer tests on sizable
texts (though not better than all of them combined), and better than all
our computer tests combined on the very short, sonnet-length passages we
used in the test. It is also, we would guess, better than three of our
four past pilot-study groups mostly of Claremont Colleges students
(below). The mean for the SHAKSPER group is barely a point and a half
short of a Bronze. For reference, our preset range boundaries were:
Golden Ear, 24-28 out of 28; Silver, 22-23; Bronze, 20-21. No one got a
tin ear, 12 or less, because chance tends to pull all scores, high and
low, toward the mean. If you don't recognize anything at all and guess
at random, you still have 50-50 odds of getting each guess right, and
you will much more likely get a 15 or a 16 than a zero. Getting a zero
would be a remarkable feat, implying powers of discrimination comparable
to what is needed to max the test with a 28. With a 35% average failure
rate, about what SHAKSPER's was, you would expect 1.1 Tin Ear (below 12)
purely by chance; we got none. You would also expect 1.1 Golden Ear
(24+) purely by chance; we got seven, and would conclude that their
success has to be more than pure luck.
Of the 24 rated players, Bronze or better, the pro gross average was
22.7, amateur average, 21.3, all rated combined, 22.0. These amount to
81%, 76%, and 79% gross accuracy, respectively. Stated differently, the
average rated pro player had a Silver Ear and could correctly ascribe
four out of five passages; the average rated amateur could get three out
of four - some of which in both cases, however, were from memory, not
from intuitive detection. Again, we know of no computer test, or
combination of computer tests, which could reach this level of accuracy
in identifying such short passages.
III. Gross accuracy tweaked upwards by aggregation and screening.
Now let's get to the tweaking. We knew from our pilot study of a dozen
Claremont students in a 2002 Shakespeare class that you could bring out
a group's latent accuracy by screening for the best and averaging for
the group. The best student in the group got 79% right; the top half of
the group, horizontally aggregated by majority rule for each question,
got 85% right; and the whole class, so aggregated, got 89% right. These
are both gross and net figures because student recognition of the
passages was close to zero. Could we do the same with a larger, more
experienced panel of SHAKSPERians? The short answer is not quite the
students' remarkable level of nine out of ten, but still an impressive
five out of six, net, after removing all recognized passages.
In our day the classic demonstration of the benefits of aggregation,
since revealingly elaborated by James Surowiecki in The Wisdom of
Crowds, 2004, was performed routinely in business school classes. The
class would be asked to guess the number of beans in a jar, and, despite
their best efforts to get it right, would find that their answers were
all over the lot, just like the law students. But, far from dismissing
it as another "humbling experience," the professor would go on to plot
the guesses. If the class took the task seriously, as they generally do
in business schools, the usual outcome was this: the guesses would form
a bell curve, with the peak of the curve within five or ten percent of
the actual number of beans in the jar. Thanks to the Wisdom of Crowds,
the class as a whole would often equal or surpass the accuracy of its
best member, and with far greater reliability and predictability, since
no one can tell in advance who is going to make the best guess. Chance
is much more of a factor with one considered guess than with many. Our
adaptation of the Wisdom of Crowds to the Golden Ear test was not just
to average takers' individual scores, but also to average their answers
to each question and draw from these "horizontal" averages a collective
group score. For the SHAKSPER takers, the two tweaks combined raised
gross group accuracy dramatically, from an untweaked individual average
accuracy of 66% to a tweaked collective average accuracy of almost 93%.
Here's how we did it:
We have seen that the gross vertical average of final individual scores
of all takers was 18.6, with 66% correct answers. The collective gross
score of all takers, first finding the group's majority horizontally on
each question and then scoring the whole group's averaged answers
vertically, just like a single test, was 22, or 79% correct answers.
Simply aggregating the whole group's guesses on each question raised
their gross accuracy from two out of three to four out of five.
As it happens, the gross average of individual scores of all Rated (that
is, Bronze or better) takers was also 22, exactly equal to the
four-out-of-five accuracy of the whole group. Aggregation raised the
whole group's gross accuracy 13% (i.e., 79% - 66%) to the same level as
its top 30% unaggregated. Aggregating the top 30%, in turn, raised its
gross accuracy another 13.9%, to 26 out of 28, 92.9% accuracy, one point
better, as it happens than the anonymous top-scoring individual. As it
further happens, gross aggregated accuracy was identical for Shakespeare
and for non-Shakespeare. Unaggregated, the top players got a gross
average of four out of five right; aggregated they got 13 out of 14
right. Both of the tweaks combined raised the group's gross collective
score from 18.6 to an astonishing 26, and its gross collective accuracy
from 66% to almost 93%, and it seems that both tweaks contributed about
equally to the improvement. After these two tweaks, the experience
might seem not quite so humbling.
IV. Net Accuracy: After removing recognized passages, still tweakable
to almost Five out of Six.
But we need at least one further tweak, and this one brings the average
back down a bit, to four out of five for the group, and five out of six
for the Top-30% Rated players. All the gross numbers discussed so far
make no allowance for recognition and treat every correct answer as if
it came from intuition. Unless recognition is zero, as it almost was
for our student pilot panels, but surely was not for our SHAKSPER panel,
this is bound to overstate the power of intuition.
We tried to avoid familiar passages, identify them, and exclude them
where found.
"Avoid" means that we tried to pick the least familiar passages,
especially Shakespeare passages, we thought we could find. Jim
Carroll's complaint (8 July) that our "Shakespearean passages have been
chosen to minimize what is distinctive about Shakespeare: his
interesting diction and his constant stretching for metaphorical
expression" probably reflects this precaution, but we thought it far
preferable to exclude archetypical passages like "Friends, Romans,
countrymen" or "Shall I compare thee to a summer's day?" from a test of
intuitive detection than to include them and either miscount them as
true detection or toss them out as obviously remembered, in which case,
you would have to wonder why we bothered to include them in the first
place. We would guess that Shakespeare's most archetypical passages are
also his most familiar, and they are more likely to be tests of memory
than of intuition. If we succeeded in choosing unfamiliar passages, the
ones we chose are no doubt less distinctive than passages randomly
selected from Shakespeare -- but the alternatives, choosing familiar
passages, or even choosing at random without regard to familiarity,
would have been too big a waste of our takers' time and our own.
Steering clear of the most familiar passages alone was enough in our
pilot studies to make sure that only one of the students recognized even
one passage. The students' net accuracy was therefore essentially the
same as their gross accuracy because, for them, almost every passage was
a case of first impression. However, as we shall see, it wasn't enough
for SHAKSPER, especially for Shakespeare passages, and we had to correct
for it.
"Identify" means that we asked takers to tell us outright whether they
recognized each passage. The responses showed us that, with a group as
sophisticated as SHAKSPER, our efforts to avoid familiar Shakespeare
passages were not always successful. Our worst choice from this
standpoint was a passage from Twelfth Night, which was recognized by 45%
of all the takers and 75% of the Rated takers. Two other play passages
and one Shakespeare sonnet got 20-30% recognition from the whole group
and 40-50% recognition from the Rated players. The other Shakespeare
questions averaged maybe 7% recognition for the whole group, 18% for the
ranked group. The overall average recognition rate was three or four
times higher for Shakespeare than for non-Shakespeare, and twice as high
for rated players as for the group as a whole. That is, on average, 15%
of the whole group and 29% of the rated group recognized our Shakespeare
passages, and 4% of the whole group, and 8% of the rated group
recognized the non-Shakespeare passages. By the same token, however,
they did not recognize 70-85% of our Shakespeare passages, and 92-96% of
our non-Shakespeare passages making these fully and properly testable by
our methods.
"Exclude" means we tried, using simplifying assumptions, to find a way
to exclude recognized passages from our accuracy estimates. To get from
gross accuracy, which is inflated by recognition, to net accuracy, which
is not, we assumed that essentially all recognition identifications
would be correct and subtracted them both from the group's total correct
answer and from its total valid takes. Spot-checking the top two
categories, who got a quarter of the recognitions, indicates that the
assumption is not quite true, only 96% true. Four percent of their
supposed recognitions were wrong. But the actuality is close enough to
100% to allow us to use it as a simplifying assumption, which would very
slightly overstate net accuracy. It is preferable to calculating the
impact of each recognition separately and manually for 2,240 answers,
and far preferable to using gross accuracy only.
This exclusion process lowered the average of individual percentages by
one to eight percent, with the greatest reductions for Shakespeare and
rated players, where recognition was very high, and the least for
non-Shakespeare and the whole group, where recognition was lower.
Overall reductions from gross averaged individual accuracy to net were
6% for the rated group and 4% for the whole group. Here is a global
summary of all our averages, with averaged individual accuracy
percentages above and aggregated group accuracy below:
Table II. Average Gross and Net Accuracy Rates, Individual and
Group
All, gross All, net Rated,
Gross Rated, net
Shakespeare 66% 60% 76%
66%
Non-Shakespeare 66% 66% 81%
80%
All 67% 63% 79%
74%
Aggregated (Group)79% 79% 93%
82%
This is our most important, bottom-line table, which gives the vertical
average of individual scores (that is, the sum of all correct answers
divided by the sum of all unrecognized takes), above, and the
horizontally, majority-rule-for-each question aggregated group score,
below, gross figures to the left, net to the right. The key figures are
now the net ones. What leaps out from it to our eye is (1) that netting
out the recognized answers, unsurprisingly, cuts the Shakespeare
accuracy percentages much more than the non-Shakespeare; (2) it narrows
the gap between the Rated players' averaged individual scores and those
of the whole group slightly, from 12 points to 11 points, thanks mostly
to lower net Shakespeare recognition, and (3) surprisingly, it cuts the
gap between the two groups' aggregated group scores from 14 percentage
points to only three. Netting for recognition made no difference at all
for the whole group's aggregated accuracy score of 79%, but for cut the
Rated group's aggregated accuracy score from a dizzying 93% to 82% --
only slightly higher than the whole group's despite the rated group's
much higher individual accuracy.
I haven't fully worked this out with Valenza, but he thinks the basic
forces at work are mathematical. For a fixed error rate the majority
decision gets better as you increase the size of the voting population,
and the convergence benefits should be less pronounced with a smaller
population, even if it is more skilled, especially if you have already
squeezed out most of the group's latent accuracy by aggregation. If you
are innocent, he notes, you would be better off in principle with a jury
of four 60%-accurate jurors and a 1% chance of conviction than with a
jury of two 80%-accurate jurors and a 4% chance of conviction. In
practice, majority rule, different levels of skill among the
test-takers, and different levels of difficulty among the passages
complicate this. Again Valenza: "If students vote on true/false
answers on a math test, and certain questions are out of the reach of
all, aggregation won't help at all on such questions, so their aggregate
hit rate will converge on the number of easy questions and stick there."
See Section VI below for some striking examples of convergence among
SHAKSPER respondents, both in getting right many passages that now look
easy, and in getting wrong a few passages that now look hard.
Whatever this says about the value of screening, it suggests that
aggregation can still boost a group's net accuracy significantly, 16
percentage points for the whole group, six for the rated group, and that
aggregated group accuracy is almost four out of five for the whole group
and almost five out of six for the rated group.
V. Other discounts: Honor system, replicability, choice of samples, and
sample size.
There were several important differences between our SHAKSPER panel and
our prior student pilot panels. Though most of the students had read or
seen several Shakespeare high school favorite plays like Julius Caesar,
their recognition of our supposedly obscure passages was much, much
lower. So were their stakes in the outcome, their eagerness to take the
test, their expectations of their own performance, and their overall
Shakespeare investments. They had nothing to lose from a low score, no
incentive to pump up their scores, and little opportunity to do so
either, since they all took the same test on paper at more of less the
same time in more or less the same place and didn't get the answers till
the tests were all in.
SHAKSPER was a different matter. Its members are heavily invested in
Shakespeare, often with a conspicuous attachment to one side or the
other of a hot debate. Many of them trust much more to their intuition
than to our stylometrics. Many are pros; a few are A-list Shakespeare
celebrities. They are highly knowledgeable and perfectionist. They
have hard-earned reputations (or at least hopes of them) which they can
trade on and don't want to jeopardize. For many issues, especially
abstract, symbolic ones or obscure, complicated, technical ones like
Stylometry, it is often reputations, more than evidence, that seem to
carry the day. One-upmanship, though less blatant and all-consuming
than on HLAS, is still very much the coin of the realm on SHAKSPER.
Many members have a stake in the outcome and much to lose from getting a
known low score. This means that the incentives not to take the test or,
taking it, not to rest content with a low score -- far less to let the
results be bruited around -- are much stronger than they were for our
students. It's hardly surprising that Shakespeare A-list grandees did
not hasten to put their name to our test; they had much to lose and
little to gain from it. And a web-based test like ours, which doesn't
take your name before the test, and which tells you the correct answers
afterward, is easier to take advantage of than a names-taken paper test
of the same people in the same room at the same time.
Anyone who offers such a test to an audience like SHAKSPER has to deal
with tradeoffs between what you have to do to get people to take the
test at all and what you have to do to keep them from giving biased or
inflated results. Many social scientists would have wanted us to build
in hard controls on bias and inflation: strictly randomize the takers;
have a control group; don't tell anyone the answers, make sure they
can't easily copy or Google the test; give everybody a name or a code
and put cookies in their computers to make sure they can't take it
twice; or, best of all, tell the ones from Canada and the UK not to take
the test and make the others all come and get tested in the same room at
the same time with a timer and a monitor present, just like the College
Board, which has excellent reasons to take such precautions for a
high-stakes test.
Most of these hard controls seem to us inappropriate for a group like
SHAKSPER, too off-putting, too impractical, too pointless, or too easy
to get around. We chose soft controls. We tried to make the test as
inviting, non-threatening, non-onerous, and rewarding as we could,
short, net-based, and with as much anonymity and feedback as anyone
could want. We tried to keep the perceived stakes as low as we could.
We limited the experiment to ten days. We asked people not to retake
the test or discuss the questions online while the test was going on.
In short, we relied heavily on the honor system and soft controls to
keep the test one of first impression.
We think we did the right thing, and believe that test abuse was close
to zero. We found only two obvious retakes, both innocent, and both
later self-identified for us by the takers. The rest (admittedly a
partly qualitative judgment) look legitimate to us. If there are a few
fudged ones, it is extremely unlikely in a test with this many takers
that they would change the outcome by more than a percent or so, or,
perhaps more important, that the change, if any, would overstate the
group's accuracy. None of the comments we have seen so far are
concerned in the least with overstating the group's accuracy; the
overwhelming concern has been with understatement.
Three such criticisms, suggested by Jim Carroll, Bob Grumman, and a
couple of offline correspondents are that the samples are too short
(Carroll and the correspondents), the test too long (Grumman), and the
Shakespeare passages insufficiently distinctive (Carroll). We have
previously discussed the last point. If you are testing for Shakespeare
detection, and not for Shakespeare recall, you want passages that not
everyone knows, and it would hardly be surprising if these were less
distinctive than, say, "Lay on, Macduff," or "my kingdom for a horse."
As for lengthening the passages, it is possible that longer passages
could be easier to identify by intuition, as they surely are by
stylometrics. But the costs of using much longer passages seem to us
prohibitive. The only practicable choices for a test like this are many
passages and short or few passages and long. "Many and long" is not a
real option because it would make the test much too long to take, even
for SHAKSPERians. Judging from just a few e-mails from takers, our
test, with 31 sonnet-length passages, takes about 15-20 minutes to
finish with snap judgments, and up to 40 minutes or so if the judgments
are more studied. Suppose we tripled or quadrupled our passage length
to make it comparable to the shortest passages you can reasonably expect
to test with computers, that is, to 500 words instead of 140. It's hard
to imagine such an expanded test taking less than an hour and easy to
imagine, not just Bob Grumman's, but every test taker's patience and,
worse, their focus, wearing thin. And would we still hear complaints
that the longer passages were also too short? We can't rule it out. It
is not wise to make your test a Marathon if you want your takers to take
it, finish it, and pay close attention to it all the way through.
It's also possible, both with SHAKSPER and with HLAS, to imagine a few
dedicated Marathoners whose patience and focus would not wear thin, and
who would not just eyeball the text, but would spend whatever time it
took to comb through the passages for tell-tale words like "Dunsinane"
or "Osric," or, worse, comb it for those deplorable, countable
stylometric tell-tales that people like us spend our days looking for,
and that intuition is supposed to make superfluous - feminine endings,
open lines, contractions, hendiadys, incongruous who's, and the like.
Worst of all, they might Google the passage; it wouldn't be hard to do.
Whatever you may think or say of such rational, left-brain deployment,
it is not intuition, and long, high-stakes passages invite much more of
it than short, lower-stakes passages.
Though shortening the test, to, say, eight long passages, instead of 31
short ones, might at first glance seem to get the test back to a
reasonable length, we would expect it, if anything, to raise the stakes
on each passage, and, with it, the temptation to study each passage
harder -- and much longer - and, again, supplement or supplant the
right-brain intuition we are trying to test with left-brain deployment,
which has nothing to do with intuition. What would our most predictable
critic, Jim Carroll, say of a test with just three or four long
Shakespeare passages? That it was just the ticket and welcome evidence
of our growing methodological sophistication, or that it, too, had grave
methodological shortcomings and was not ready for prime time - with far
too few passages for a fair test of Shakespeare or anyone else, yet
still was too much of a Marathon to ask of reasonable takers? There are
many other serious problems with few-but-long passages - they are less
broadly representative and, hence, more subject to variability; they are
much more vulnerable to recognition; it's much harder to find ones that
aren't; the distortion costs of using one like our unfortunate Twelfth
Night passage that everyone turns out to know is much higher; and you
would have a lot of explaining to do if you tried to make
apples-to-apples comparisons of irreparably short passages like Shall I
die? with longer ones -- but we think the ones we have discussed should
be enough to make our case. We don't pretend to have said the last word
on this subject, and, as always, we invite others to try different
tradeoffs than the ones we used, but nothing is either good or bad but
alternatives make it so, and neither lengthening the passages nor
lengthening the test strikes us much of an improvement.
VI. Identification Hits.
As with our student panel, most of our SHAKSPER answers to each
question, Shakespeare or non-Shakespeare, right or wrong, showed very
high intra-group agreement as to whether or not the passage was by
Shakespeare, and also showed high agreement between the whole group and
the Rated group. No more than 7% of the aggregated answers look like
tossups. The other 93-96% show majorities of 57% or more. If numbers
like these were reported in a national election, everyone would consider
it a landslide (Table III). I call this consensus; Valenza calls it
convergence.
Table III. Group Consensus: Very High, but Not Always Correct
All figures net accuracy
Shakespeare
Non-Shakespeare All
High consensus, questions answered correctly
Full panel 9 (68-80% maj) 11 (59-100% maj)
20 (59-100% maj)
Rated only 11 (64-88% maj) 12 (57-100% maj)
23 (57-100% maj)
High consensus, questions answered incorrectly
Full panel 3 (64-67% maj) 3 (57-69% maj)
6 (57-69% maj)
Rated only 2 (73-78% maj) 2 (57-58% maj)
4 (57-78% maj)
Tossups, all incorrect
Full panel 2 (51% maj) 0
2 (51%)
Rated only 1 (53% maj 0
1 (53%)
No tossups were correct
This means that both panels had high consensus on 26 or 27 out of the 28
questions and were closely divided on only one or two. Looking at
high-consensus answers only, the full panel got 20 of 26 (77%) firmly
right, in gross, and the other six firmly wrong. The Rated panel got 23
of 27 firmly right (85%) and the other four firmly wrong. We'll skip
the details of the impressive Shakespeare "firmly rights" and go
straight to the equally impressive Non-Shakespeare "firmly rights."
Table IV shows that neither panel had much trouble with most
non-Shakespeare authors represented.
Table IV. Eleven Non-Shakespeare Hits
Passage Percentages who thought it non-Shakespeare (full
panel/rated only)
Listed in declining order of Rated percentages.
All percentages net.
Bacon poems 87/100%
Middleton 89/100%
Chapman 82/100%
Spenser 78/96%
Fletcher 67/91%
Daniel 75/87%
Marlowe II 71/83%
Shall I Die? 65/82%
Earl of Oxford 59/79%
Marlowe I 70/78%
Jonson 60/72%
All of these seem like solid hits to us, both according to what we see
as the orthodox consensus and according to what our computer evidence
has done to confirm it. None of these tested passages seem likely to be
Shakespeare's. Not everyone agrees with us or the orthodox consensus on
every passage, but the important point here is that remarkably few of
our test-takers thought these passages sounded like Shakespeare. We
would hardly consider numbers like these a humbling outcome for the
group that produced them. Shall I Die? was the only one of these widely
recognized (by 31%/54% of the two panels), but few of those who did not
recognize it thought it was Shakespeare's. Would longer passages
greatly enhance these landslides? We doubt it; they are already so
lopsided it's hard to imagine longer passages changing things much, even
if they should be easier to identify. Do they signal that the group is
befuddled by too-short passages? It doesn't look like it.
VII. Identification Misses.
However, two outcomes, though equally convergent and consensual, were
not so impressive for accuracy, and a third showed the full group at
odds with the Rated group (Table V).
Table V. Two and a half Non-Shakespeare misses
Passage Percentages who thought it non-Shakespeare (Full
panel/Rated)
All percentages net
Oldcastle 31/42%
Drayton 43/43%
Funeral Elegy 41/57%
The Oldcastle and Drayton passages, one recalling a beleaguered-stag
scene from As You Like It, the other a sonnet from Drayton's Idea,
suggest that even strong majorities of both groups can be fooled by
well-turned, vivid, image-rich passages by other writers. It is also
possible that Drayton, who Henslowe says co-authored the play Sir John
Oldcastle (1600) with Anthony Munday, Robert Wilson, and Richard
Hathway, could have written both of the confounding passages. A second
edition of Oldcastle ascribed it to Shakespeare, and it was included in
the 1664 Folio and Brooke's Apocrypha, but we know of no one today who
seriously ascribes it to Shakespeare, and our tests say it's very
unlikely to be Shakespeare's work (our 2004, p. 402). No takers,
incidentally, recognized the Oldcastle passage and only one recognized
the one from Idea.
What about the Funeral Elegy? Donald Foster relied in part on computer
tests to prove that the Elegy "couldn't not be Shakespeare," and he did
speak of intuitive "sniff tests" with a hint of disdain. When Brian
Vickers' crushing counter case, Counterfeiting Shakespeare (2002)
loomed, and Foster abandoned his Shakespeare ascription, the dull,
pious, pedestrian Elegy of the eye instantly became Exhibit A for those
who say you should always trust your gut instincts, never anyone's
computers.
If so, and if the whole SHAKSPER group's intuitions were to be taken as
the only valid test, Foster should have stuck to his guns on authorship
and reconsidered his disdain for sniff tests. Only 6% of the whole
panel recognized our Elegy passage, and 59% of those who didn't thought
it was Shakespeare's! On the other hand, a net 57% of the Rated Panel
thought it was not Shakespeare. Our tests say that Foster did the right
thing to concede, and that the Elegy is on a different statistical
planet from Shakespeare, though it could easily be by Ford (our 2001).
Maybe it's only the best ears we are supposed to listen to, but it's not
easy to know in advance which ones those are on any given passage, and
they are not always connected to the best mouths. On balance, the
conflict between rated ears and all ears does little to make the Elegy
look like a reliable success story for detection by gut instinct.
Table VI shows two Shakespeare misses for both panels, and two equivocal
tossups.
Table VI. Two Shakespeare misses and two more divided panels
Passage Percentages who thought it Shakespeare (Full
panel/Rated)
All percentages net
The Rape of Lucrece 38/22%
Pericles Act V 33/47%
Love's Labor's Lost 49/64%
Venus and Adonis 49/59%
Only one taker recognized our passage from The Rape of Lucrece, and very
few of the others thought it was Shakespeare's, fewest of all, oddly, on
the Rated panel, which was otherwise generally more accurate than the
whole group. Seven takers recognized our passage from Pericles, Act. V,
but two-thirds of the whole panel, and 53% of the Rated panel, thought
it was not Shakespeare's. Pericles is generally considered co-authored
by Shakespeare and George Wilkins; scholarly consensus gives Acts 3-5 to
Shakespeare, and our tests agree with it.
The passages from Love's Labor's Lost and Venus and Adonis were
recognized by 19 and three takers, respectively; for the others, the
whole group was divided half and half, but clear majorities of the Rated
group correctly ascribed them both to Shakespeare. It's not clear how
to score these. Two misses and two tossups? Or two misses and two
half-hits? Neither seems an unequivocal success story for these
identifications.
VIII. Three shots in the dark.
Three passages on the test were not scored, since scholarly consensus as
to who wrote them is not settled. But we tested them anyway in case we
found the group's instincts helpful in determining actual ascriptions.
This is wholly uncharted territory, but, if we had a computer test that
looked like it might be 80% accurate, we might not bet a thousand pounds
on it, as we have on some of our computer tests, but we certainly would
not want to let it sit on the shelf unexplored. The same may be said
for ascription by gut instinct. With tweaking, it can reach 82% group
net accuracy for passages of known authorship, and we can't imagine
SHAKSPERians not being curious as to what it says about passages of
unsettled authorship. Table VII gives the outcomes:
Table VII. SHAKSPER's Group Ascriptions for Three Doubtful Passages
Passage Percentages who thought it Shakespeare (Full
panel/Rated)
All percentages net
1H6 87/89%
A Lover's Complaint 42/26%
Edward III 68/73%
14% and 25% of the panels recognized another beleaguered stag scene from
1H6, Talbot before Bordeaux. 87/89% of those who did not recognize it
thought it was Shakespeare, one of the most lopsided majorities on the
test. Gary Taylor assigns the scene, 4.02, to Shakespeare; Paul Vincent
thinks it is co-authored by Shakespeare and "Author Y." Marcus Dahl
could find no hand but Shakespeare's in the whole play. Our own tests
are ambivalent on the scene as a whole. It looks like a Shakespeare
could-be by all our regular tests, but an improbable by one new test.
The passage itself is much too short for our tests. We lean toward
Vincent's view of the whole scene, but the stylometric evidence is hard
to judge - much harder, it seems than the intuitive evidence.
SHAKSPER's judgment in this case is consistent with all four views of
1H6, though one could argue that SHAKSPER's judgment on non-Shakespeare
beleaguered-stag passages is not terribly reliable .
Perhaps surprisingly, only one person recognized the passage from A
Lover's Complaint. Could it be more discussed these days than read? Of
the many who did not recognize it, 58/74% thought it was not
Shakespeare. MacDonald Jackson, Kenneth Muir, and most scholars of the
late twentieth century have assigned LC to Shakespeare. Our best guess
(our 1997 and 2004), and Brian Vickers'(his 2007), and Marina
Tarlinskaja's is that it is not. SHAKSPER's judgment favors the
doubters, but, again, one could argue that SHAKSPER's group judgments on
Shakespeare poems outside the Sonnets don't look like its most reliable.
Five and eight percent of the two panels recognized the Countess Scene
passage from Edward III, which we take to be a recent addition to the
consensus Canon. Our tests say it could be Shakespeare, and 68/73% of
the two panels seem to agree.
IX. How SHAKSPER's Round 1 compares with other takers.
If we were to seek a kind of "control group" to compare with SHAKSPER's
takers, we would turn to several Claremont student groups who have taken
the test, or a precursor, or we would look to our computer tests. Four
student groups have taken this, or a previous test: Valenza's
preceptorial class of entering freshmen in 1995; the Claremont Rugby
Side on a bus in New Zealand in 2002; Ann Meyer's Class on Shakespeare's
Tragedies in 2002; and a sprinkling of Philosophy, Politics, and
Economics alumni volunteers in 2007. We have also, over the years,
identified a couple of miscellaneous Golden Ears from casual takers, and
we have gone for advice to a small group of pros, most notably MacDonald
Jackson and John Farrell, but also Brian Vickers, Lisa Hopkins and Matt
Steggle, none of whom should be deemed in any way responsible for any of
our test's shortcomings . None of our tested groups got the extensive
recordkeeping and analysis that we have given the SHAKSPER groups, but
it is safe to say from memory that SHAKSPER outperformed the
preceptorials, the Rugby Side, and the PPE students.
Whether it outperformed the Claremont Shakespeare class is not so clear.
We no longer have the students' individual scores, nor their
individual averages, but they were the least casual, and the
most-studied of our pilot groups. Here is an adaptation of a posting we
sent to SHAKSPER in 2004:
Table VIII. A Gross-Score Comparison of SHAKSPER with Claremont Pilot Group
Claremont Students SHAKSPER
2002
2007
Worst individual: 54% correct, gross 50% correct,
gross
Best individual 79% correct, gross 89% gross
Best combined 84% correct, gross, n=6 93% correct,
gross, n=24
All combined 89% correct, gross, n=12 79% correct,
gross, n=80
18 of the student group's 25 successful identifications (72%) were by
lopsided votes, two-to-one or higher. 16 of the whole SHAKSPER group's
22 successful identifications (73%) were lopsided in this sense, as were
20 of the Rated group's 24 successful identifications (83%).
A better comparison, since SHAKSPERians, on average, recognized about a
tenth of the passages and the students next to none, would be between
the students' gross scores, essentially equivalent to their net scores,
and SHAKSPER's net scores. Such a comparison would look like Table IX:
Table IX. Net-Score Comparison of SHAKSPER with Claremont Pilot Group
Claremont Students SHAKSPER
2002 2007
Worst individual: 54% correct, net 44% correct, net
Best individual 79% correct, net 87% correct, net
Best combined 84% correct, net, n=6 82%, correct, net, n=24
All combined 89% correct, net, n=12 79%, correct, net, n=80
It would not be surprising if a group of 80 got higher-highest and
lower-lowest scores than a group of 12. It's mildly surprising that the
smaller, far less knowledgeable, less motivated group got higher group
accuracy than the larger, more knowledgeable, more motivated group, but
the smaller the group, the more likely is its outcome to be a fluke.
Recall, for example, that the Shakespeare class did much better than the
other three Claremont student groups tested. Could that have been a
fluke? Perhaps it was also mildly surprising that the whole of the
student group, aggregated, did better than the top half, aggregated,
which, in turn, did better than the best individual. For the students,
aggregation was a bigger boost to accuracy than screening. If so, the
reverse SHAKSPER outcome, where the best individual surpassed the best
group, which surpassed the whole group, would not be a surprise. For
SHAKSPER, screening seems to have been a more powerful tweak than
aggregation. There must be a literature on this, perhaps to be gleaned
from James Surowiecki's footnotes, but we have not studied it. For now,
we would say that the two sets of results are remarkably similar and we
would guess, where they differ, that the larger SHAKSPER panels give a
better notion than the student panel of what you can reasonably expect
do with intuition and what you cannot.
It would be possible, with more manual counting than we want to do now,
to compare the net accuracy of Shakespeare pros with those of amateurs,
and to see whether literary critics did any better or worse than stage
performers, artists, scientists, and so on. It's all buried in the data
and retrievable in principle, but it would not be easy, and it is not at
the top of our agenda. We would guess from the similar accuracy levels
of SHAKSPER and the student group, and especially from the surprisingly
similar gross accuracy levels of SHAKSPER's own pro and amateur
respondents, 68% and 66%, respectively, that the difference in pros' and
amateurs' net accuracy, if any, would be barely detectable, and not
necessarily favorable to the pros, whom we would expect to recognize
more passages than amateurs. Table X also shows a remarkably narrow
range of average gross accuracy scores for the various subgroups identified:
Table X. Gross Accuracy Scores of Identified Subgroups
Subgroup Number Gross Average Gross Average Accuracy
Score %
Professionals 31 18.9 67.5
Amateurs 49 18.6 66.4
Critics 26 18.7 66.8
Writers 14 18.9 67.5
Artists 33 18.6 66.4
Other 21 18.0 64.3
Some of the categories overlap. "Other" is mostly people who declined
to state a category.
Would net accuracy differ greatly from these gross accuracy scores? We
don't know, but it seems improbable.
One might imagine that writers and artists would be more intuitive, and
critics more analytical (see Simonton, Origins of Genius, 1999), but the
average accuracy of the three categories is virtually identical. We did
not ask anyone to list their IQ's or their verbal and math SAT scores,
considering it too nosy and off-putting even for us, but we would have
loved to have had them, and maybe their college majors as well. Four of
our best six students in the 2002 pilot study were science majors; the
two best of all were science majors from Harvard-bright Harvey Mudd
College; the others were from our college, Claremont McKenna College,
whose students, on average, are only Columbia-bright. We have to wonder
whether any of our SHAKSPER takers were scientists.
Another way of looking at such questions on a smaller scale might be to
look at the top end only, Golden and Silver Ears, where we would expect
recognition to be at its greatest. Conveniently, there are seven Golden
Ears, six of them pros, and six Silver Ears, five of them amateurs. The
86%-pro Golden Ears said they recognized, on average, a remarkable 27%
of all questions; the 83%-amateur Silver Ears claimed to have recognized
14% of all questions, half the Golden Ears' rate, but more that twice
the rate of the other five-sixths of the group, which was 6%. Golden
and Silver ears are only a sixth of the whole group, but the big
difference between the two top groups, one mostly pro, the other mostly
amateur, is fully consistent with our commonsense guess that pros would
recognize more passages than amateurs.
If so, we can roughly calculate the two top groups' net accuracy, as
explained in Section IV above, simply by subtracting recognitions from
both correct answers and total takes of each question, giving us net
correct answers as a fraction of net takes. This cuts average
Golden-Ear accuracy from 87% gross to 81% net, and Silver-Ear accuracy
from 79% gross to 75% net. With recognized passages removed, both
groups got fewer passages right in fewer questions, with the
high-recognition Golden Ears, not surprisingly, losing more accuracy
than the lower-recognition Silver Ears. Netting for recognition, in
effect, reduces Golden Ears to Silver, and Silver to Bronze, with the
top pros, on average, still retaining a 6% detection accuracy edge over
the top amateurs, with 81% net accuracy, versus 75%. Their gross edge
had been 8%, 87% versus 79%.
Suppose we sought an aggregated, majority-rule on each question group
score for the Golden Ears only? It's not clear that they would score
much higher than the rated group as a whole (82%), which, let us recall,
was only three points higher than the whole group (79%). All 7 Golden
Ears recognized four of the 28 passages. For the remaining 24 passages,
where there was some unrecognition to be had, there was a net correct
majority on 20, a net incorrect majority on three, and a 50-50 tossup on
one. If you put aside the tossup altogether, that would give the Golden
Ears 20 right out of 23 (87%). If you give half-credit for the tossup,
they get 20.5 right out of 24 (85%). If you count the tossup, but give
no credit for it, they get 20 right out of 24 (83%). Any of these would
be arguable, but we would consider the most conservative of them, 83%,
the most defensible.
We would conclude from this that the Golden Ears were far ahead of all
others in recognition of passages, and 6% better than the Silver Ears in
net intuitive detections of unrecognized passages, but horizontal
aggregation doesn't seem to boost group accuracy as much at the highest
level as it does for the whole group. Unlike the Claremont students in
the pilot study, the best individuals in the SHAKSPER group did better
than the whole group aggregated, and better than the best of the group
aggregated. Several of these individuals also did better than the best
Claremont student, even after netting out recognized passages. Net
group accuracy for Golden Ears could be five out of six (83%), but it
would take a bit of indulgence to get it to six out of seven (86%). Top
pros look better than top amateurs at recognizing passages, and it is
probable that this is also so of all pros, through we haven't tested for
it. We have very little evidence that any category of taker surpasses
the others on average.
X. How the SHAKSPER group compares in accuracy with stylometric tests.
Here is what we said about the Claremont pilot study in 2004:
How does this accuracy compare with that of our best quantitative tests?
"Far higher" would be a persuasive answer for such short,
Sonnet-length samples. All of our quantitative tests are sensitive to
sample length because longer samples average out more variance than
shorter ones, giving us tighter ranges and higher discrimination for
long samples than for short. Most of the samples we used in our Golden
Ear test have no more than 150 words, far shorter than any for which we
have dared to validate any of our quantitative tests. For comparison,
our current estimated composite accuracy rates for longer,
single-authored passages look something like this:
Text Shakespeare Non-Shakespeare
Whole plays 100% 100%
Poems, 3000 words 100% 100%
Play Verse, 3000 words 95% 100%
Poems, 1500 words 100% 100%
Play Verse, 1500 words 96% 88%
Poems 750 words 93% 71%
Play Verse 750 words 97% 75%
Poems, 470 words 92% 73%
Not much has changed since then. Our accuracy figures remain the same,
and the SHAKSPER Golden Ear outcomes are similar to those from our
student pilot study, only slightly higher for the best individuals, but
somewhat lower for the group. SHAKSPER's double-tweaked accuracy from
intuition alone is not as good as our tests have been on samples of
1,500 words or more, but it's in the same ball park with our accuracy
for passages of 470 to 750 words - and it is far higher than we would
expect any or all of our tests to do on the very short passages tested,
which averaged about 140 words. If Golden Ear had been tried as a
stylometric test, it probably would not quite have met our test criteria
- of around 95% reliability in saying "could be" to known Shakespeare,
and at least 20% reliability is saying "couldn't be" to known
non-Shakespeare, but that is because we rely on negative evidence and
are far less tolerant of false negatives than of false positives.
Without question, intuition is far better than all our tests combined on
the sonnet-length passages we tested.
As we explained in Section V, it is conceivable that using longer
passages would raise Golden-Ear accuracy, but doubtful that anyone could
devise practicable intuitive tests for, say, the 1,500-word or
3,000-word passages for which we consider our stylometric tests to be
well validated. If we, or anyone else, could find and offer 28
3,000-word passages for identification, the test would be equal in
length to Hamlet, Macbeth, Romeo and Juliet, and The Comedy of Errors
combined and would take more than a day just to read, let alone analyze,
in entirety. In general, the longer the passages, the fewer can be
tested without expecting miracles of motivation. From this perspective,
Golden-Ear testing may be almost as impractical for wholesale testing of
long passages as computers are for testing short passages.
XI. Conclusions
In sum, after much tweaking and netting, SHAKSPERians as a group seem
capable of getting almost four out of five, or five out of six,
identifications right, and its very best individual did a bit better
than that, with net accuracy reaching 87%. However, only two of 80 test
takers got net accuracy higher than 83%. The average Golden Ear had 80%
net accuracy, silver 75%; the average individual SHAKSPERian taker had
63% net accuracy. SHAKSPER's overall performance roughly matched the
best of our student pilot groups, with aggregated group accuracy
slightly lower, and the accuracy of the very top individuals somewhat
higher. SHAKSPER's recognition rates were much higher than those of the
Claremont pilot group, and its Golden Ears' rates much higher than the
rest of SHAKSPER. No one else has taken the test as seriously as
SHAKSPER. None of the subcategories of takers stood out as much better
or worse than the others, and the differences in gross accuracy between
pros and amateurs seem remarkably small. Intuition seems much more
accurate than stylometrics for very short, sonnet-length passages;
stylometric seems more accurate than intuition for longer passages, but
an actual head-to-head comparison of many long passages seems impracticable.
XII. Golden Ear Round 2
Golden Ear Round 1 has given us what looks like a highly talented panel
of a dozen rated, identified SHAKSPERians, to take Golden Ear Round 2.
We haven't yet asked them whether any of them want their names or their
particulars, pro/amateur, writer, player, etc., disclosed to SHAKSPER or
anyone else. To these we might add up to eight or nine rated players
discovered in previous tests, if we can find them and get them to serve.
A couple of unrated SHAKSPERians want to take Round 2, and we would
not grudge them the experience, nor would we be above retroactively
including some or all of the dozen rated players who did not identify
themselves on the test, or, for that matter, other unrated players who
want another shot, but we need enough identification on Round 2 to
e-mail it to the proper recipients and to relate their Round 2 outcomes
to their Round 1 outcomes.
Unfortunately, we don't have Round 2 on the web and shall have to find a
way to mail or e-mail it to takers to do by hand, and to be scored by
hand. Like Round 1, Round 2 will have a known, scorable component, both
to help calibrate the test, which we would guess is more difficult than
Round 1, and to provide more data points to help see how much, if any,
of the high accuracy rates found at the top of Round 1 was luck of the
draw. It's one thing to find and congratulate the best guesser of the
number of beans in the Business School jar; it is quite another to
expect him to do it twice. Round 2 will also have some of its own shots
in the dark, passages whose authorship is not settled.
XIII. A last note on methodology
In some ways, it is astonishing, given the frequency and fervency of
declarations that intuition can outperform Stylometry - or that
styometry can outperform "sniff tests" -- that no one has ever tried to
see whether, and to what extent either proposition is actually so. We
have been trying to remedy that for twelve years and have now, at last,
thanks to the help of our student programmer, Ryan Wilson, and his
advisor, Arthur Lee, gotten a Round 1 survey up on the net and gotten an
excellent response from SHAKSPER, which has permitted us, at last, to
try a first cut at an answer. We have used the best methodology we
could manage, but who are we to proclaim that the tradeoffs we chose in
our first big outing are the ones that should bind all others? For an
exercise as novel as this, it would be surprising if further
experimentation with different parameters did not produce new insights
perhaps wiser and more penetrating than ours, and informed by our
mistakes, as well as by our successes. Every adventure is a
reconnaissance for the next, and this seems to us a question begging to
be explored from more than one perspective. As always, if anyone in or
out of SHAKSPER would like to try an experiment with different tradeoffs
than the ones we chose, and SHAKSPERians were willing to take it, we
would be happy to help them out. In the meantime, we consider our
tradeoffs reasonable ones and our evidence the best currently available.
We hope it will inspire better.
In the meantime, we would like, again, to thank the SHAKSPERians and
others who took our survey for giving it their full attention, and
especially for honoring our request to withhold their online comments
till the test was over, so as not to wreck the test for others. We now
welcome comments, online and off, but we hope the online ones will take
care not to give away too many specifics of the test, which cost twelve
years of pretesting and hundreds of dollars for programming to prepare,
and, now, many hours from SHAKSPERians to take and us to analyze. We
would just as soon keep it available for future use with different
groups, and as a standard against which other versions can be measured.
We hope that SHAKSPERians will help us keep it so, as much as possible.
This e-mail address is being protected from spambots. You need JavaScript enabled to view it
(the address, not the professor) was retired July
1, 2007; please use
This e-mail address is being protected from spambots. You need JavaScript enabled to view it
instead.
Ward Elliott
Burnet C. Wohlford Professor of American Political Institutions
Claremont McKenna College
Pitzer Hall, 850 Columbia Ave.
Claremont, CA 91711-6420
(909) 607-3649
Fax (909) 621-8419
This e-mail address is being protected from spambots. You need JavaScript enabled to view it
http://govt.cmc.edu/welliott
"Better grey words with crimson examples than crimson words with grey
examples."
_______________________________________________________________
S H A K S P E R: The Global Shakespeare Discussion List
Hardy M. Cook,
This e-mail address is being protected from spambots. You need JavaScript enabled to view it
The S H A K S P E R Web Site <http://www.shaksper.net>
DISCLAIMER: Although SHAKSPER is a moderated discussion list, the
opinions expressed on it are the sole property of the poster, and the
editor assumes no responsibility for them.
|
