Survey FAQ

Foreword and Summary of Results

A Smallville survey was conducted through a sister site earlier this year (2006), starting at the beginning of March and running for a few months in several stages.  In fact additional segments may still be done.  The survey drew from a diverse, four-figure pool of potential respondents, a pool that wasn't limited to core Smallville viewers or frequent posters.  For a large percentage of the survey, individual invitations were sent to potential respondents randomly selected from the pool.  The invitations provided a unique user name and password to access the survey page and care was taken to minimize question bias.  In other words, lots of work went into this, with the objective being meaningful results!  Here are some key numbers (where they conflict with the FAQ detail, it's because the following summary has the latest updated results).

On the character front, Clark, Lex and Chloe are the three most popular characters on the show, all having scores of 8.6/10 or slightly higher.  Lana is the least popular, with an average score of about 4.9/10, and Lois has the next lowest score at about 6.1/10.

In the Clark romantic relationship categories, the results parallel the individual rankings of the three female characters.  Clark-Chloe leads with an average score of about 7.25/10, Clark-Lois is second with a score of about 5.75/10, and Clark-Lana trails with a score of about 4.6/10.  The relationship scores are lower because of strategic voting, where some respondents try to drive down the competition's rating, and in Chloe's case for a couple of other reasons -- "Chloe's too good for Clark," and "Nice twist but 'THEY' will never do it".  Those latter factors actually reflect underlying stronger not weaker support for Clark-Chloe.  In the case of Lois and Clark-Lois, there's also an iconic bias that acts in the opposite direction (i.e., pumping up the score).

Virtually everyone -- 90% to 95% -- rates at least one of the above three relationships 7/10 or higher.  Or, looking at it another way, those who favor no ship or don't care are in a very small minority.  Clark-Chloe, especially after considering the Clark-Lois iconic bias (which places that ship closer to Clark-Lana), is by far the consensus ship among viewers.  Over 50% give it a 9/10 or 10/10, over 70% give it a 7/10 or higher, and just under 15% score it 2/10 or lower.  Of that 15% negative, almost all of it is strategic voting or one of the other two factors described in the last paragraph.  True strong opposition is less than 5% and possibly less than 2%.

Support for the "general rule" that Clark should end up with Lois is high -- 7.75/10 -- but the vast majority who score the rule highly also score Clark-Chloe high, as high or even higher.  There is no inconsistency here; the respondents are saying "yes, the general rule is good, but the exception here is better."

Since the survey was conducted over several months it's interesting to point out some of the key trends in results as the season went on.  The Clark-Chloe score increased while the other two ships decreased.  Lionel Luthor became slightly more popular, his individual score crossing the 8/10 mark in the latter segments of the survey and putting him in an even clearer fourth spot.  Finally, the Lex-Lana or "Lexana" ship moved up from fourth spot to third, i.e. it's now running slightly higher than Clark-Lana or "Clana".  An evolution of viewer opinion as the season progressed is one explanation of these changes.  However, another is that more casual viewers (who took longer to respond to survey invitations) were inclined to rate a bit differently than the initial wave of respondents.  So they may have rated Clark-Chloe higher, for example.

The above results will not necessarily parallel other online polls, which have heavy self selection biases and often significant question bias and other problems as well.

Now to the FAQ-style Table of Contents. 

Survey FAQ - Table of Contents

(Relatively More) Survey-Related Questions:

1.  Great... Another Online Survey... You Know It's Meaningless, Right?

2.  Self-Selection Bias... Non-Representative Samples... Ring a Bell, Dude?

3.  Okay, I'll Bite... What Was That TV Guide Online Poll All About?

4.  You Say Only One Segment of Your Survey Was Openly Posted?

5.  What's the Margin of Error on this Survey?

6.  It's Still Mostly an Online Survey and Onliners are Non-Representative, No?

(Relatively More) Smallville-Specific Questions

7.  So Should The Powers That Be Abandon "Lexana"?

8.  What's the Survey Say About Chloe/"Chlark" and Lois/"Clois"?

9.  What's That "Chlois" Theory About?

10.  Which Smallville Seasons Are Most/Least Popular?

11.  Given Smallville's Renewal, What's the Survey Say About Season 6?

---------------------

(Relatively More) Survey-Related Questions:

1.  Great... Another Online Survey... You Know It's Meaningless, Right?

No, your Friendly Neighborhood Webmaster ("FNW"!) isn't in the habit of conducting surveys he knows are meaningless.  But thanks for the thumbs down, before knowing much if anything about it, or us.

Back to Contents

2.  Self-Selection Bias... Non-Representative Samples... Ring a Bell, Dude?

Sure those ring a bell, and you know something about polling, sampling, margins of error, survey bias and so on.  But it's been your FNW's experience that many people who know something about those things think they know more than they do.  They don't necessarily understand the concepts.

For example, a self-selection bias exists to some degree in virtually every survey or poll.  Yes, it's greater when an open call to participate is posted online, which only one segment of this Smallville survey (the Usenet segment) did by the way.  You also have to guard against multiple voting in polls like that.  But even randomly-selected invitees in a scientific poll, contacted by phone or in email, decide whether to participate or not.  They self-select in a similar way.  In openly posted polls, the potential for self-selection bias is greater but can vary widely depending on all the circumstances.  A recent Smallville TV Guide Online poll was an interesting case.

Back to Contents

3.  Okay, I'll Bite... What Was That TV Guide Online Poll All About?

It was a fascinating Smallville relationship poll that illustrated the effect self-selection and non-representative sampling can have. Chloe won the poll by a more than  2-1 margin over Lois, with Lana trailing far behind.  We'll skip the detailed analysis we had here in this section before, but the bottom line is that Chloe's margin of victory was probably somewhat overstated, while Lana's results were probably understated.

Despite its flaws, the poll wasn't meaningless though because you had to register as a user with TV Guide Online to take the poll, and after you voted once the system wouldn't let you vote again.  So right there were some controls you don't see in all polls, which would tend to make it more reliable.  Being a general site like TV Guide, as opposed to say a site that advocates a particular relationship pairing on Smallville, also argues in favor of the poll's worth.  Again, you have to look at all the factors when assessing the worth of a poll or survey.

Back to Contents

4.  You Say Only One Segment of Your Survey Was Openly Posted?

As regards Smallville, yes.  It was an invitation crossposted to 5 Usenet groups where Smallville was on-topic.  It drew a total of 49 respondents, probably 5-7 of whom had a strong self-selection bias because I posted the invitation.  It's a long story and you'd have to know Usenet.  In any case they voted 0/10 for a particular question or two, and 10/10 for another.  It was expected, and I was trying to draw out whatever was there, while also fire-testing the questions and the form.  Usenet is a very contentious place, and useful as a test of fire.  The Usenet segment was also interesting for purposes of comparison.  On the character ratings, there's no indication of any major skew, just the more critical Usenet bias, meaning every character tends to get lower rated on average (see later for the reverse phenomenon over at Devoted to Smallville, on most questions at least). Most Usenet respondents voted normally though, and we separate out the Usenet and non-Usenet results of the survey.

Later segments of the survey had no pure self-selection bias.  The next several segments drew from non-Smallville-specific cyberspace.  There was a Superman component in one case.  That was a mailing to those who had participated in the Superman movie casting survey (it drew 1,100+ respondents at the time).  Tom Welling won that survey, so there is some Welling bias in our survey on Smallville, for that segment.  Clark Kent's score is probably a bit higher than it otherwise would be in the survey as a whole.

There was also a mailing to an Enterprise survey list, and posted surveys to three Lost groups on the web for example, where they were invited to fill out a Lost survey and were then asked to do a second one on Smallville.  They didn't know that Smallville one would be there, so it's akin to an email invitation in a way.  Yes, there could have been a self-selection bias introduced here in theory, but it's unlikely and if there were it was probably not significant.

The most recent segment of the survey, the Random Smallville Cyberspace segment, covers several web discussion boards, for the most part proportionately represented in the survey based on number of posters.  Basically, a census of poster names was done for several episode discussion threads on each site, not Clana or Chlois or Tom Welling threads or the like.  Episode discussion threads are likely to have a more diverse pool of posters, not just one-issue posters.

An alphabetical list of posters by site was then used as the sample pool.  The first "A" was marked to receive an invitation and then every fourth poster name after that.  If a poster was "No E-Mail" in the site's e-mail system, or had already been invited on another site (this was a manual check based on eyeballing identical poster names) then the next poster on the list was marked and so on.  The result is that 1/4 of the sample pool receives invitations, with individual user names and passwords, across multiple sites.

Point being, a lot of work was put into making the survey as meaningful as possible and eliminating bias.  Moreover, it was NOT a survey of frequent posters.  Someone who posted once to a Smallville board, and who might have had more posts to other show boards, had an equal chance of getting selected as the person who made dozens of posts.

Back to Contents

5.  What's the Margin of Error on this Survey?

Even the concept of Margin of Error would only be relevant here to the non-Usenet segment of the poll.  On the non-Usenet segment, it's not an exact analogy but it is possible to think of the "out of 10 scale", and a 5/10 average rating for a particular question, as a flip of the coin where either 0/10 or 10/10 comes up.  Of course the votes are spread out more than that when you're rating characters and not flipping coins, but a scenario where half the votes come in at 0/10 and half at 10/10 would yield the 5/10 average.  Yet another way of looking at it derives from the stat we report on "positive" scores, for example, which we categorize as 8/10 or better.  If we say 50% rate a character positively, it's akin to a political poll where 50% say they have a favorable (or very favorable) opinion of a candidate.

So very roughly, these out-of-10 scores can be thought of as having some connection to the margins of error on political polls or coin flips.  Most Margin of Error tables are based on a 95% confidence level (the "19 times out of 20" part), and assume a 50% incidence of whatever is being measured.  So they're basically coin flip tables, except with a more normal distribution to get to the 50-50 average.  If you randomly sample 400 from a population, you'll have a +/- 5% margin of error with 95% confidence.  There are different margins of error for lower populations and incidences other than 50%, but that's basically the idea.

"Ah, so you'll need to have had 400 respondents to the non-Usenet part of your survey, in order for the Margin of Error to be +/- 5%.  So if character A has an out of 10 average score of 5.0, it could be anywhere from 4.5 to 5.5?"  Allowing for the fact that we're doing comparisons here that are somewhat apples and oranges, yes that's conceptually about it.  But 400 respondents (we don't have that many yet but it is well on the way there) are only required to get a 5% Margin of Error at a 95% confidence level.  We're just doing a survey here on Smallville.  Relax.  We don't need or attempt to get 95% confidence on a particularly low margin of error.  If, in a particular segment we have 132 respondents (as we do in some demos), there's a 75% chance (i.e., level of confidence) that we got within 5% even on that small sample.  There's a calculator here that lets you play around with these numbers:

http://www.raosoft.com/samplesize.html

The point is you can get close with smaller samples.  There's no magic in the "19 times out of 20" or 95% confidence level; it's just a convention or common property of most surveys you see.  It still means 5% of the time you'll be outside the margin of error, but -- if you understand the concepts here -- you know the likelihood of being that much outside gets progressively lower quickly.  You might be 6% out instead of 5%.  So what?  There's no magic between being out 5% and not a smidgen more.   The single most likely result is that you got it almost exactly right, and you will have got it almost exactly right some of the time.  This is another thing people don't fully get about margin of error.  They'll focus on the +/- 3% or whatever as if that was a likely error.  It's not, in fact it's right at the border of that rare 1 in 20 times exception, where the confidence level breaks down.  The likelihood of the poll being out by the margin of error is actually very low.  It's much more likely closer than that.

So if Chloe is up at 8.5 and in a three-way race for first with Clark and Lex, and Lana is at 5.25 in last, the margin of error calculations are different but basically you know with virtual certainty that Chloe, Clark and Lex are much more popular than Lana.

Back to Contents

6.  It's Still Mostly an Online Survey and Onliners are Non-Representative, No?

It is mostly online, yes.  We did do a few interviews offline but those are also onliners in a very real sense.  The vast majority of people have online access these days, and especially in the CW's target demo of 18-34 almost all are online.  The main bias risk is in the very frequent, very noisy posters to Smallville cyberspace.  The person who posts once to a Smallville episode thread, perhaps with a question or quick comment, on a web site or discussion group with diverse rather than narrow focus, is probably not that much different, on average, than the broader viewership in terms of Smallville opinion.  In some segments of the survey -- and we may do more of these -- we even poll completely different target groups and then ask about Smallville after the other subject.  So they have no clue they'll be asked about Smallville until after they take the "main" survey from their point of view, and we effectively draw respondents from outside Smallville cyberspace that way.  Yes, they're online because almost everyone is these days, but they aren't Smallville junkies if you must think of everyone posting in Smallville cyberspace that way.

Which you shouldn't, because getting back to the Random Smallville Cyberspace segment again, the sample pool on that will be in the thousands, i.e., 2,000+, and only one post or very infrequent posts can get you in that pool.  This is a difficult concept for some frequent onliners talking it up on web boards, with perhaps a few dozen other frequent posters whose names they recognize.  Their "world" is awash in frequent posts from a few dozen, often very vocal about their preferences.  Yet frequent posters are only a small percentage of Smallville cyberspace, and the same with any other show.  The survey draws from a four-figure pool of many posting names the frequent posters rarely see and might only vaguely recognize, if at all.

One of the sites we've done is Devoted to Smallville, a site that conventional wisdom would say is one that favors Lana and Clana.  That's how the site started out at least.  It's since had "Devoted to..." Tom Welling, Allison Mack, Erica Durance, Michael Rosenbaum and other actors and shows.  So it's a more diverse pool than just Lana or Clana.  The main obvious "skew" of the results that we saw is that they rate almost everything higher than the non-DTS respondents, EXCEPT for Lex-Lana or "Lexana" which they rate a whopping 2+ points lower than the non-DTS respondents (e.g., 3 instead of 5/10).  Yes, the margin of error here is high on a small segment, but it's much more likely than not that there really is a significant difference here.  Why?

Well, the interpretation of those results should be easy.  "Devoted to..." is what the respondents at that site are.  As a general rule they give higher ratings.  In fact in the individual ratings Chloe finished second to Clark on that site, the only two to both be over 9/10,  higher than their overall average in the survey, which is in the 8.5/10 range.  But DTS still has the Lana and Clana-centric origins, if you will, and the recent turn Smallville has taken with Lana and Lex is not popular.  It's akin to how the Chloe and Chlark fans would feel about a Chloe-Lex pairing.  The so-called "Chlex" has its fans, but there would also be massive opposition to it as a serious arc.  The Lois & Clark TV series in the 90s almost didn't get renewed after season 1, and had several creative changes made.  One of the elements that many viewers disliked, potential and lapsed viewers included, was the Lex-Lois romance business in season 1.  Of course all such comparisons in Smallville are made more complicated by the fact we have one Lois in name, and the "real" Lois (Chloe, Daily Planet reporter) in all but name.

Back to Contents

(Relatively More) Smallville-Specific Questions:

7.  So Should The Powers That Be Abandon "Lexana"?

[The question's premise comes from the low Lexana score in the Devoted to Smallville segment, as described in the last part of question 6]

No, I don't think so at all.  In the early segments of the survey, after "Cyborg", there was massive indifference to Lex-Lana with some strong opposition.  No one was really giving it 9s or 10s, probably because no one knew for sure what to think about it.  Many noted the better acting chemistry between Kreuk and Rosenbaum, in #100 Reckoning and also Lockdown before that, but romantic Lexana (this question is in the romance pairing section of the survey) they weren't sure about.

Towards the end of the season, we began to see 10/10 scores for Lexana, which were virtually non-existent before, and more 8s and 9s on the Lexana question as well.  It may only be 20 percent but more viewers are loving it so give it some time. Consider also the positive (8-10), neutral (3-7) and negative (0-2) ratings of the three young female characters.  Rounded off only a point or two, in the most reliable segments of the survey those ratings are:

Chloe : 80% positive, 15% neutral, 5% negative

Lois:  50% positive, 30% neutral, 20% negative

Lana:  33% positive, 33% neutral, 33% negative

Lana is a character that desperately needs a new direction at this point in the series, and in Lex's camp is the obvious place for her to be.  TPTB can't go back to romantic Clana again, without damaging and possibly destroying the show and, even worse, the opportunity for greatness it still very much has.  Lana is now "the one that got away" as Smallville's Executive Producer Al Gough described it in season 1.

Back to Contents

8.  What's the Survey Say About Chloe/"Chlark" and Lois/"Clois"?

Well, for starters let's get back to just the individual Chloe and Lois ratings, positive and negative as shown in the last section.  Those were:

Chloe : 80% positive, 15% neutral, 5% negative

Lois:  50% positive, 30% neutral, 20% negative

In the out-of-10 ratings, the three most popular characters, in all segments of the survey, are Clark, Lex and Chloe.  In fact that's the case in every sub-segment except the DTS one where they put Lex in fifth behind Martha and Jonathan.  That's the same phenomenon as the anti-Lexana sentiment there.  Lex gets "hammered" for that a bit, but I put "hammered" in quotes because Lex is still rated 8.44/10 at DTS.  That's very much in the range of Lex's overall rating in the survey.  Clark, Lex and Chloe are all up in that 8.5/10 area.  They're close enough that it's too close to call between them.

Among the romantic pairings, Clark-Chloe or "Chlark" leads.  It leads by less than a point, Chlark in the high 6.xx/10 while Clois is in the low 6.xx/10.  But Clois has an iconic bias as does Lois individually, and a couple of other things have also shown up that illustrate how perilous interpreting these results can be.  Many people rate more than one relationship highly, and some strategic vote, trying to drive down one ship's score while raising up their favorite.  So it's tough to get a true picture of how much real, hard-core opposition there is to Chlark.  More than 50% rate Chlark very highly (9/10 or 10/10), and 70%+ rate it 7/10 or better, in what's perceived as a three way race remember. So we know there's strong support on the positive side, but what about the negative?   It's often cited by a few frequent posters on a few boards that (paraphrasing) "They can't do Chlois because there'd be a fan revolt" or the like.

Not only is there absolutely no evidence that there is any critical mass for a revolt, anywhere in cyberspace (even fanboy sites like aicn.com favor Chloe and Chlois), it's the exact opposite and the survey indicates this just with the raw scores.  Less than 10% of respondents who wanted Clark-Lois, did not also rate Clark-Chloe at least somewhat favorably.  Of those less than 10%, some were strategic voting, possibly as many as half.  A few, including via comments, even show evidence of a "Clark doesn't deserve Chloe.  He deserves Durance's Lois, and vice versa" way of looking at it.  That "Clark doesn't deserve Chloe" sentiment has been seen online many times.  It's not a go signal for the writers to do Clois, it's a go signal for the writers to improve Clark's character in this area, and indeed that's what they seemed to do in the season 5 finale scenes between Clark and Chloe, including the dialogue.

Getting back to the negatives on Chlark, on the surface they're in the 10%-15% range.  But after factoring in strategic voting and the other factors noted, true strong opposition is less than 5% and possibly less than 2%.  That's among Smallville viewers of course, so the show does have a communication and selling job -- an opportunity really -- to convey the Smallville Twist on the story to a broader audience.

Back to Contents

9.  What's That "Chlois" Theory About?

The survey never mentions the word Chlois.  Few outside core online fandom have even heard of the word.  But virtually everyone who's watched Smallville knows the ambiguity, if you will, i.e., that Chloe is the young Lois Lane character in all but name.  Check out the rest of this site for more on that.

Back to Contents

10.  Which Smallville Seasons Are Most/Least Popular?

This question has a lapsed viewer bias and is very prone to incorrect interpretation.  Basically, several million viewers voted by clicking their remotes on Smallville towards the end of season 2, and into seasons 3 and 4.  They stopped watching the show in massive numbers.  We can't know for sure, but it's a reasonable deduction they didn't leave because they loved those seasons and couldn't stand watching a 10/10 show.  Most likely the show was starting to bore them, or they were getting annoyed with the soap opera element, and so they found something better to do.

The flip side of that is that the people who stuck around are likely to, on average, have had a higher tolerance or affinity for those seasons.  Viewers with that higher tolerance (for what many considered not good enough TV) were the ones around to vote in the survey at the end of season 5.

That said, season 4 is the clear loser of the race even without adjusting its score lower to correct for the bias.  So it's safe to say season 4 is the least favorite, more than a point behind all the other seasons.  Those other four seasons are in a close race in raw score, meaning Season 3 is the next least favorite after adjusting for the lapsed viewer bias.  Beyond that it's tough to say.  In the early part of the survey it looked like season 5 would be #1 for a time, but the latter part of season 5 has dragged down its average considerably.  What's left is a too-close-to-call race between seasons 1, 2 and 5.  You decide.

Back to Contents

11.  Given Smallville's Renewal, What's the Survey Say About Season 6?

We didn't ask any season 6 questions per se, but the results suggest several things about what would and wouldn't be a good idea at this point.  More Clana, bad idea.  Static Lana, bad idea.  Change is needed there, probably following through with Lexana and seeing where it goes.  Static Lois, also bad idea.  The character is the second lowest rated, and worse she continues to weigh down one of the three most popular characters, Chloe.  Give her an alternate destiny and she could become hugely popular.  I don't favor it, but considering the ratings problem the show faces a Death of Lois Lane event in season 6, perhaps February sweeps, also has to be considered.

Chlark AND Chlois, good ideas but as a practical matter Chlois is the much more important of the two.  You can't have constant Chlark for two seasons, but nor can you sell "Chloe's gonna die in the series finale to drive the final wedge between Clark and Lex and make way for Lois".  Tragedy buffs love it but loyal viewers of a Superman show don't.  Big Picture Vision, with the Smallville/Chlois twist in particular, is central to the destiny of this series.  It's the difference between what's potentially the best Superman-related  incarnation ever (if they do other things right as well), versus an Ultimate Contrivance fiasco that makes a few tragedy buffs go "Oooo".

Smallville Season 6 debuts on the new CW in the U.S., on Thursday, September 28, 2006.  Tell everyone you know to be there.  :-)  The show needs it.

Back to Contents

Webmaster