Sins of omission? An exploratory evaluation of Wikipedia’s topical coverage
Alex Halavais & Derek Lackaff
Looking at authority/authoritativeness of wikipedia articles. Accuracy is obviously an issue, but authority is more than that. It’s also about breadth, timeliness, etc. Encyclopedias are thought of as being authoritative, but what about the Wikipedia.
Halavais and Lackaff took three encyclopedias and compared entries in the encyclopedia with entries in the Wikipedia to measure breadth of coverage.
- Lerner & Trigg (eds.) Encyclopedia of Physics
- Preminger and Brogan (eds.) New Princeton Encyclopedia of Poetry and Poetics
- Stranzy, P. (ed.). Encyclopedia of Linguistics
Each encyclopedia has its own way of organizing and naming topics, so it’s difficult to find an exact match of topics between encyclopedias.
Between 18% (poetry) and 37% (physics) of articles found in the print encyclopedias were not found in the Wikipedia, but it still does have a large number of the articles (a majority). Wikipedia covers the physical and natural sciences much better than the humanities. However, the physics encyclopedia covered more general topics while the poetry encyclopedia covered very specific topics (stanzas, iambic pentameter, etc.). These different approaches to covering topics affect the results as well (making their numbers kind of useless), but it is very true that the sciences are covered much more than humanities. Many of the topics in the Wikipedia cover the topics in the poetry encyclopedia, but they are part of more general articles rather than having their own specific topic. The sub-topics can easily be found because Wikipedia is searchable.
Why is this useful? According to Halavais, the study identifies where wikipedia is weak and needs to be better developed. But does it really? I don’t think these things can really be compared. Also, how the models differ for organizing and presenting topics.
Wikipedia presents a new model of authority. What is it? They never explain that. I suppose they mean the “peer review”/collective intelligence idea, but they never really discuss this.
I frankly don’t think this study of theirs looked at authority at all, but it did make the important point that it’s difficult to compare coverage. Two encyclopedias can cover the same things, but one may have general articles that cover more topics and others will have many more specific articles that cover very specific articles briefly.
The really important question to me is: How do you define authority?
Halavais was talking about how librarians see the Wikipedia in black or white — as either good or bad. I disagree and I think it is really an unfair characterization (yeah, all librarians think exactly the same way, right?). I think there are three issues that librarians are dealing with. 1) is it authoritative (and what does that mean)? 2) should it be used by students in their research? and 3) how is it being used? I think the answer to #1 is “in some cases” and the answer to #2 is “it depends.” It’s one thing to read the Wikipedia to get background info on a subject with the knowledge that the information may not be 100% accurate. It’s another thing to take a Wikipedia article completely “at it’s word” and use the information in a paper without looking for corroborating evidence. Perhaps part of the measurement of authority should consider what the article is being used for?
I get seriously annoyed with these negative characterizations of librarians as being completely against the Wikipedia. How about we’re critical? We’re skeptical. Which is what everyone should be. Grrr! More stereotypes. And Alex Halavais used to work at Buffalo’s School of Informatics!
Comparing Wikipedia and Britannica: The story behind Nature’s encyclopaedia story
by Jim Giles
Jim did the Nature study comparing the Wikipedia and Britannica. They used 50 pairs of articles — one from Wikipedia, one from Britannica.
Asked experts to evaluate two articles in one topic in a “blind study”, looking for inaccuracies and misleading statements. They were looking at the articles from the POV of the reader to see if the information would mislead him or her (which is a bit different than factual inaccuracies).
Results: 3.9 errors/article in Wikipedia. 2.9 errors/article in Britannica. Very few major errors in either source. More detailed data analysis was not performed — it was a journalistic article, not a scholarly study.
In March 2006, Britannica wrote a rebuttal (PDF) saying that almost everything in Nature’s investigation was wrong and misleading. They also demanded a retraction.
Criticisms of the article included:
- referee reports were not given to other sources – protecting their anonymity
- some of the content came from the Britannica Student Encyclopedias – when doing a search on Britannica.com material from both sources come up, and, furthermore, is it ok to have inaccuracies in a student edition???
- Britannica material was edited – in a few cases, Nature combined two articles — like Ethanol and uses of ethanol — to make the article comparable to the Wikipedia article. They didn’t change content/facts
- reviewers made mistakes – Nature assumed that the experts were right, but it was a comparative exercise and the same expert evaluated both articles so they were equally likely to make a mistake with either source.
Thoughts on evaluating Wikipedia.
- Is Britannica the best benchmark?
- how should omissions be dealt with? Is omission as important inaccuracy? It’s certainly an issue.
- How important is timeliness?
- Can style/clarity be evaluated?
What I got out of this is that the idea of authority is a really murky one and should take much more than accuracy into account.
I think the ‘Wikipedia problem’ will be with us for some time to come. As a school librarian I often agonise over what to tell my students about Wikipedia. We subscribe to an online encyclopaedia, but it is hard work persuading people to use it – they tend to go straight to Wikipedia, which they already know and which doesn’t require them to log in. I personally think that Wikipedia can be a very useful resource, so I don’t want to tell students not to use it, but teaching them the skills to use it critically rather than passively is difficult. That said, it could be argued that Wikipedia only forces issues into focus which are already there. After all, I’d like my students to approach all their information sources critically.
I know several school librarians who have developed interesting and creative lessons using Wikipedia as an introduction to the idea that information should always be scrutinised and cross-checked. One approach I particularly liked was mentioned to me by someone at the School Library Association’s conference (UK) recently – she said she told her students always to check the discussion page of Wikipedia articles in order to get a sense of whether the information was disputed in any way. In this sense Wikipedia arguably has the edge over other information sources, where debate may have taken place invisibly.
Overall, I feel Wikipedia reflects the larger issues of information literacy today. Once upon a time it was possible to give students (particularly school students) a limited number of ‘safe’ resources. Now the library is more abstract concept, there is less control over what goes in and who writes it, so critical skills are more important than ever.
And yes, the librarian stereotypes bug me too! Although I was pleased today when our schools ‘Learning to Learn ‘ programme coordinator commented that he could have run the programme without the help of the Library. I guess if we all keep chipping way on our home ground, the stereotype will shift in the end!
Meredith,
Thanks for the reports. I hope to attend your half-day session at Internet Librarian.
Rick
You’re right, of course. The good librarians see the world in an amazing rainbow of grays. I come from a library school, and have a high opinion of librarians. I don’t think I made a blanket statement, but if so, it was unintended.
But…
I was picking on school librarians in particular, because I’ve run into so many of them who clearly say “Wikipedia isn’t written by experts so it’s no good.” Those are the librarians who grab the headlines and make you wonder what they are thinking.
As for the issue of “authority,” I agree. We do not offer a new definition of authority. What we have suggested here is that it is important to understand how the content here differs from the traditionally defined authoritative sources.
Wikimania 2006: “According to Wikipedia…”…
…I now regret not having gone to “Sins of Omission?” and “Comparing Wikipedia and Britannica,” but check out Ethan Zuckerman’s “Omission and Accuracy?” and Meredith Farkas’s “On Accuracy and Authority” for the content……
In response to Alex’s comment above… I have to pitch and and defend school librarians. It is true that there are a lot of people in the school library world who are quick to condemn Wikipedia, but there are plenty of people who have a more balanced response to it (as I mentioned in my earlier comment). The problem, though, it that it can be very hard to get kids to treat any information critically, so it is tempting to just say ‘use a “proper” encyclopedia – anyone could have put information on Wikipedia’. Also, I know a number of school librarians in the UK who are facing SEVERE funding cuts which are justified by school administrations on the grounds that the internet – and Wikipedia in particular – are all that students need. I think those situations make people hostile (I have no idea how the situation in America compares – school libraries seem to have more status generally over there). It IS frustrating, though, to hear people who really should know better dismissing Wikipedia so completely. Personally, I welcome new tools and resources.
Personally, I have no idea why people would not understand how to do this.
Besides the issues of accuracy, authoritativeness and critical skills, it is also an issue of economics.
I visit Wikipedia very often simply because they offer free information. Of course, I’m quite on the alert as to the page content and writing style/correctness but who wouldn’t be irked visiting Britannica and Encarta/MSN when just about everything you really need is a “premium article”.
Sorry, but I don’t have the funds to subscribe to their services.