Who’s the Boss? The difficulties of identifying censorship in an environment with distributed oversight A large-scale comparison of Wikipedia China with Hudong and Baidu Baike

In 2008, Baidu’s chief scientist William Chang said “There’s, in fact, no reason for China to use Wikipedia . . . It’s very natural for China to make its own products.” Today Hudong (baike.com) and Baidu Baike (baike.baidu.com) greatly eclipse the Chinese-language version of Wikipedia despite (or because of) the censorship known to take place on the sites. However, identifying outright instances or patterns in censorship can be difficult due to the (mostly) user-generated nature and oversight of the content. Instead, this project attempts to perform a large-scale comparison of the three services, matching thousands of Chinese-language Wikipedia articles with their in-China counterparts, in order to identify the “content gaps” in the two baike (Chinese for “encyclopedia,” which we use to refer to Hudong’s and Baidu’s online encyclopedias). Censorship—or at the very least anomalies in the generation of content—might be identified by articles that don’t exist, “protected” articles that are not editable by regular users, and by articles that are much shorter than those on Wikipedia China. The reason might  is emphasized is due to the distributed oversight nature of these online encyclopedias, where not only governments but also companies and users get to play the role of content gatekeeper. This decentralization makes attributing who is responsible for apparent censorship more difficult, a topic which this report will explore in detail by examining how it functions in these online encyclopedias.

In addition to the exploring the difficulties in identfying censorship, this post will also lay out the research methodology of the article matching, some of the initial results, and the next steps to be taken in the coming months as we continue to analyze this data and expand the project. As the data for this project has just been collected, this report is more data dump than fully-composed, critical analysis (see the final section for future avenues of research and thinking), but hopefully it serves as an introduction into the sorts of quality data available in this project.

Tables with lists of articles that are protected/locked on Wikipedia, Baidu Baike, and Hudong as well as a list of articles that are found on Wikipedia China but not found on the two baike are below. You can jump directly to them by clicking the links in the previous sentence, but as the data is still preliminary and has many limitations, one might be best served reading through the following sections.

Baike: Chinese encyclopedias

As William Chang of Baidu foretold, mainland Chinese netizens have gravitated toward local products such as Hudong and Baidu Baike, leaving Wikipedia China to be edited and read primarily by users in Taiwan, Hong Kong, and the rest of the Chinese diaspora. Today, in terms of raw visitors and article count, Hudong and Baidu Baike dwarf Wikipedia China, which has roughly 700,000 articles versus over 5 million in each of the two baike. Certainly, while China’s sporadic blocking of access to Wikipedia at various points over the past ten years has certainly been a factor in limiting Wikipedia China’s growth among mainland users, Baidu and Hudong’s dominance may be more credited to Baidu’s entrenched position as the dominant search engine in China (thus allowing for cross-site “partnerships” and synergies¹) and Hudong’s bevy of features built into its custom wiki and social networking platform.

Though the baike are incredible sources of information on China, they have been dogged by allegations that they have liberally “borrowed” content from other websites, including Wikipedia–a not damnable offense in and of itself since Wikipedia’s content is free to share and re-use, but the two baike are for-profit and inform users that any content contributed is property of Hudong and Baidu. In the past, Baidu was noted as the worst offender, plagiarizing from not only Wikipedia without credit but also from Hudong.² Though the goal of this project is not to analyze these plagiarization claims and to quantify the amount of shared content among the three encyclopedias, the data generated from the matching of articles as explained in the methodology section below would allow someone to easily perform this sort of follow-up analysis once the data from this project is cleaned up and released.

The difficulties of identifying censorship in an environment with distributed oversight

This project began with a question: everyone “knows” Hudong and Baidu Baike, like all Chinese websites, have to restrict certain kinds of content on their websites, but is it possible to empirically prove that censorship is taking place on the sites? Thinking about different ways of testing our assumption with the publicly available data on the websites drove this project.

First, what would we consider signs of censorship on Hudong and Baidu Baike? The most obvious would be the lack of certain articles on topics that are known to be notable. Thus, using Wikipedia China as a control, we can propose that if an article on say 上海帮 (The Shanghai Gang) exists on Wikipedia China, barring censorship, it should exist on Hudong and Baidu Baike, especially since Hudong and Baidu Baike have a much larger library of entries. However, attributing the lack of an entry due to censorship is not a perfect science since Wikipedia China itself isn’t a perfect control–though entries in Wikipedia China are assumed to be of interest to Hudong and Baidu Baike users, and thus should have articles in those encyclopedias, one should keep in mind that Wikipedia China does tend to have a Taiwanese bend due to its userbase. However, using missing articles as a potential indicator for possible censorship–especially if the article that is missing is a long one–is a reasonable start.

Second, comparisons could also be made between the length of the articles between the encyclopedias. For instance, an article might exist on all three services, but they might be drastically shorter than their Wikipedia counterpart. For instance, the main body of the Wikipedia entry for 艾未未 (Ai Weiwei) is over 20,000 characters long (spaces removed) while the Baidu entry for him clocks in at 2,000 characters and the Hudong one at 3,500–this is despite the fact that article lengths for the Baidu articles sampled in this project are on the whole longer than Wikipedia’s: the mean Wikipedia China article length is 8,174 characters (median: 4,207) while Baidu’s is 12,679 (median: 9,029). Discrepancies of the sort in the Ai Weiwei article might simply be a case of greater interest in the topic outside mainland China than within, or, again, it might be another potential indicator of censorship.

Third, some especially sensitive or controversial articles are unable to be edited except by users with much higher privileges than ordinary members. Being unable to change an article is not in and of itself a sign of censorship; for instance, Wikipedia “protects” certain articles to prevent vandalism and pointless back-and-forth “edit wars.” Baidu and Hudong no doubt have similar intentions in mind as well, but what matters here is matter of transparency–while Wikipedia publishes a list of all protected pages, as far as we can tell, no such corresponding list exists for Hudong and Baidu Baike–and authority. Was it the choice of editors and users at Hudong and Baidu to classify certain articles as locked or was the decision made higher? Was there an open discussion of such matters or was a list handed down from somewhere above? Interviewing regular users of Baidu Baike and Hudong might provide insight into such questions, but for now, we have some data to start with.

Finally, there are more subtle ways to disrupt access to information, many of which we will no doubt uncover as we continue to sift through the data. One that we’ve noticed is the “failure” by Hudong and Baidu to redirect certain article titles the same way that Wikipedia does. For instance, a search for 艾神, a laudatory nickname meaning “God Ai” for Ai Weiwei, properly redirects to the Wikipedia article for him. Hudong and Baidu Baike don’t perform such re-directs. Again, whether this is a conscious decision or merely an inadvertent one cannot be answered by looking at this one example. However, by looking at such cases in the aggregate one might be able to make a more legitimate claim that something might be going on.

Many new media outlets such as Sina Weibo privilege users with the ability to generate the content that goes on the website–in essence, to be not only their own programmer or broadcaster but also the producer. However, because such websites host their users’ content, they are also in charge of regulating and ensuring that such content complies with all Chinese laws–regardless of how vague such regulations might be. Thus attributing censorship that takes place on these sites can be unclear–is it the government that mandated certain topics are off-limits or is it the company that restricts the content?–an intentional feature of the decentralized system of information control that Chinese authorities have developed.

Censorship is further distributed on the baike because now not only are users their own programmer and producer, but they also serve in an oversight capacity as an editor. Unlike Wikipedia, users who aren’t registered can’t begin editing and creating articles, but for the most part, registered users can edit most general articles, and as they engage with the site longer, they achieve greater and greater levels of ability to edit and oversee the website. Thus, there could are always at least three potential reasons for why an article doesn’t exist, an article is shorter, or an article is locked on Hudong or Baidu Baike: government entities, private companies, or users themselves. Judging whether or not these factors are genuine instances of governmental censorship or due to explainable, organic reasons can be quite tricky. Because of the multiple layers of oversight, what may appear to be outright censorship may be a less malicious (though no less pernicious) case of self-censorship.

Methodology

As mentioned, this project is a large-scale attempt to quickly and automatically match thousands of Wikipedia China articles with their corresponding Hudong and Baidu Baike entry. A script was developed which did just that, taking a keyword, locating the correct Wikipedia article and scraping the desired data, and then repeating the process for Hudong and Baidu, respectively, before moving on to the next keyword–essentially relying on Hudong and Baidu to perform the proper title matching or, if the title did not match exactly, redirect to the appropriate article.

Zhichun Wang, Juanzi Li, Zhigang Wang, and others at the Computer Science Department at Tsinghua University have already made great strides in “knowledge linking” between English Wikipedia and Hudong/Baidu Baike, and their approach to matching articles across different languages using machine learning to read semantic information is well beyond the scope of this report. However, techniques from their work may be employed in the future, though the current title matching approach employed for this projecy is already a fairly robust solution as is–especially since we are dealing with only one language. In “Cross-lingual knowledge linking across wiki knowledge bases,” Wang et al. reported that simple title matching gave them a precision percentage in excess of 99% even when having to translate from Chinese to English; however the recall rate was an atrocious 32%.

The data collected for this project still has to be evaluated properly, but thus far results seem very promising. Out of 5,143 keywords tested, Hudong and and Baidu each successfully found or redirected us to an appropriate article 3,539 and 3,600 times respectively, a recall rate of at least 68%–a number which is likely much higher once we account for the fact that the script also attempted to use search results to identify and match instances when the keyword was found in the returned search results’ snippets (the article titles and summaries).

Of these 3,500+ times, the exact title–meaning a character for character match–was made between Wikipedia and Hudong 2,037 times, and 2,080 times between Wikipeda and Baidu Baike. If one assumes that an exact match in the article title is a match in the article content as well, then the precision rating for matched articles is nearly 60% already, and that’s not yet taking into account all the articles titles which are only slightly different between the services, either due to different styling conventions (e.g. Wikipedia’s article on national universities is titled with traditional characters [國立大學] while Hudong and Baidu both use simplified [国立大学]) or other minor variations. Overall, the matching procedure, at least for the terms tested thus far, seems more than acceptable for the goals of this project, though of course attempts will be made to verify the precision and increase the recall rate.

As a test of the script and the methodology, a sample of 5,143 keywords involving a mix of topics known to be sensitive and those known to be popular were generated from five different sources: the titles of articles protected by Wikipedia (282); article titles from the list of Wikipedia China articles on GreatFire.org’s watchlist of censored Wikipedia pages (489); article titles of the top 1000 most viewed articles on Wikipedia China in April 2013 (995 after a few Wikipedia meta-pages were removed); the titles of articles that generated more than a total of 10 combined views on August 1, 0:00-1:00 and 12:00-13:00 (3,470); and finally, the only non-Wikipedia source, a list of keywords from the website Blocked on Weibo previously confirmed to have been prevented from returning search results on Sina Weibo (840). Some of the final sample of keywords appeared on multiple lists, and the overlap of four of the sources is shown below (the source of words from August 1 were dropped because 5-way Venn diagrams are not as pretty…).

The following variables were scraped from all three encyclopedias: length of article, URL, title of the article (which can differ from the URL), locked/protected status, last modified date, and a number of other meta variables. From Hudong and Baidu, other variables included the number of “likes” (that is positive votes that the page was “helpful”) and number of edits. Variables unique to Baidu were pageviews for the article, and whether or not the page contained multiple entries (unlike Wikipedia, Baidu Baike doesn’t serve up “disambiguation” pages when the user searches for a broad term which could refer to multiple topics, and instead serves up all the entries on a single page; see for instance this entry for “Jordan” which includes the basketball player Michael Jordan as well as the country Jordan). Variables unique to Hudong were number of editors and the error message given when trying to edit a locked page–either 对不起，您的题目中含有不当内容，请检查? (“Sorry, the topic contains inappropriate content, please check?”);  您不能编辑敏感词条 (“You are not allowed to edit sensitive entries”);  您不能编辑普通敏感词条 (“You are not allowed to edit general sensitive entries”).

Protected, missing, and censored(?) content

As mentioned previously, an article that is protected or locked doesn’t necessarily mean that it is being censored–Wikipedia often uses article protection as a means to prevent malicious behavior. However, the locking down of articles itself can be abused and used in a malicious manner, especially if the decision to do so is non-transparent, arbitrary, and/or doesn’t reflect the sentiment of the group. In such cases, Wikipedia fortunately has active “talk pages” which allow all users to debate and contest such issues. Baidu Baike also has discussion pages which allow users to to something similar: for instance, here’s one for the entry on the United Kingdom wherein users quibble about whether or not England can lay claim to being the place where the first “bourgeois revolution” took place (the consensus was no, the Dutch Revolt came before it). However, many protected Baidu Baike entries contain no such talk pages. For instance, trying to reach Xi Jinping’s talk page returns an error message. Hudong eschews separate talk pages in favor of comments at the bottom of each entry. However, again, oftentimes for sensitive and/or protected entries, there is no opportunity to leave your comment; the comments box simply doesn’t appear.

Furthermore, while the list of pages protected by Wikipedia is published by the site, as far as we can tell, no such list exists for Hudong or Baidu Baike. Thus, even with he caveats noted above about how the articles in the following three tables don’t necessarily conclude that censorship is taking place in those entries, this exercise still seems like a useful one if only to make such information publicly available. Again, proper analysis is yet to be done on what is protected/locked and what sorts of patterns exist in the three different encyclopedias approaches to article protection.

Note: the machine translations come from Google Translate. Those that have been corrected or verified have notes or a dot in the third column; if the third column is empty, the translation is still to be confirmed–an ongoing part of this project. Please do not disseminate unconfirmed machine translations without first verifying that they are correct.

This last table displays a list of all the nine characters or less (or four double-byte Chinese characters) keywords which successfully matched to an entry on Wikipedia, but when searched on Hudong or Baidu Baike, returned no matching results. As mentioned previously, the matching algorithm is still extremely simple, so some of these keywords may indeed have appropriate entries on Hudong or Baidu Baike. For instance, the table reports that the movie 戀夏500日 (500 Days of Summer) is not found on Baidu or Hudong. However, some manual digging uncovers that the movie is listed under the title 《和莎莫的500天》 and the Wikipedia one must be a Taiwan or Hong Kong-specific one. Thus, the below will no doubt contain false positives for terms it failed to find due to flawed searching on our part. Thus, the below is merely an initial stab at trying to uncover the content gaps in the two baike and will be refined in the coming months.

The first column is the keywords used to test, the third is the title of the Wikipedia article that was found (which may sometimes vary from the initial keyword depending on how Wikipedia interprets the keyword and handles the redirecting to what it thinks is the proper article), and the last two columns indicate whether or not that Baidu or Hudong located a matching article for the keyword.

Not surprisingly, a number of the entries appear to be related to Taiwan, which makes sense since the largest share of Wikipedia China readers and contributors come from Taiwan while Hudong and Baidu are primarily mainland China users. Thus, a number of Taiwan-related articles in Wikipedia China may not to considered noteworthy to mainland users, explaining why they are not found in Hudong or Baidu. However, others are more clearly omitted for political or prurient reasons.

The same caveat about the machine translations from above applies here. Do not disseminate the machine translations without first verifying if they are correct. We are in the process of confirming the translations in the coming weeks and months.

Next steps

As the matching mechanism appears to work, expanding the keyword list is necessary to start building a larger database for analysis. However, at some point, as the keywords become ever more obscure and less common, we’ll no doubt see diminishing returns after expanding the keyword list too far.

Using the other variables we’ve collected will also make for interesting analysis later on. For example, models could be created to predict what kinds of articles are most likely to be protected, or for spotting articles that have abnormally small numbers of edits and article lengths. A discussion of what users are trying to express when they vote a particular entry as helpful would also be worthwhile. For instance, the most liked article on Hudong was one for a banking executive while the top ones on Baidu are mostly pop-related: Jacky Wu, Michael Jackson, the Japanese manga Detective Conan, singer Yang Mi, Jay Chou, and… Mao Zedong, with over 236,000 thumbs ups–though all of them were blown out of the water by the Korean group 2NE1, whose Baidu Baike article was liked nearly 2 million times. What are users trying to say when they mark something as helpful by clicking the thumbs up icon on each baike service? For instance, what can we conclude from the fact that more people have classified Bo Xilai’s entry on Baidu and Hudong as “useful” than they have for Xi Jinping’s?  Should we think of “likes” for particular pages as an alternative way netizens can actively or passively signal their endorsement of a topic’s existence or article’s sentiment even if the content is drastically different from a more “neutral” source like Wikipedia?

By looking not only at what content and data doesn’t exist on baike, but also at the content that does, this project will investigate what knowledge and information is fit for public display. If articles are shorter on Hudong and Baidu, what information do they carry? Does this information reveal anything about the authors’ intentions? By examining the topics and articles that are left visible in these baike and considering the motivations behind those who seek out, view, edit, and approve of these articles, this project hopes to offer a more nuanced view of the typical narratives about censorship in China. Trying to understand what sorts of expressions netizens are making via these online encyclopedias, despite whatever censorship might be taking place, is as interesting as the potential censorship itself. This project will hopefully push us to once again consider the many complexities when discussing information control in environments where oversight of content has been decentralized to companies and users–an environment which makes it increasingly harder to identify traditional instances of censorship.

1. If you search for most general topics on Baidu, links to Baidu images, search, videos, shopping, bulletin boards, answers, and its own encyclopedia are among the top results. For example, here’s a search for 手机 (cell phone): Baidu links make up more than half of the first page of results, including a hit to Baike. When you search for celebrities and political figures, Baidu Baike is again always among the top results, if not the very first one. Hudong is often not found on the first page of results–if at all. Here’s a search for Xi Jinping. Notice the message at the top in bold (根据相关法律法规和政策，部分搜索结果未予显示。) stating that the results have been filtered. Those that are returned are all state media links (Xinhua, People’s Daily, China Daily, CNTV/CCTV)–except for Baike which inherits the top spot. Even for searches of people that are not filtered, Baike still almost always gets top billing and Hudong is often relegated to later in the results–though often not due to any failure on Hudong’s part. For example, the same search for 周恩来 (Zhou Enlai) on Google has both Hudong and Baidu Baike in the top three results (Wikipedia garners the top spot). All this is to say that while giving preference to one’s own services is not some unprecedented online tactic (see the outcry in 2010 when Google tweaked its searches to prioritize YouTube and other internal Google-owned content above other comparable websites), it does seem like there is something inherently baked into Baidu’s search algorithm that either intentionally or unintentionally penalizes Hudong’s articles compared to Baidu Baike and others. However, more rigorous examination is needed before a conclusion about how real or serious that penalty can be made.

2. For example, here’s an article on cloud seeding that was featured on Hudong on August 7, 2013. It’s section on the costs of cloud seeding are copied word for word in Baidu Baike on the same topic (starts with “一种是用飞机把干冰等冷却剂撒播到云中”). If you look at the version histories, the Hudong version that contains this section predates Baidu by nearly four years. For Wikipedia copying, take a look at each respective encyclopedia’s entry on 利玛窦 (Matteo Ricci). Here’s a comparison of what is shared between Wikipedia and Hudong today, and Wikipedia and Baidu Baike today. As you can see from the matching bits, Wikipedia and Hudong share much greater overlap, and if you go through the historical records, you can identify that Hudong added the majority of such material in the May 11, 2008 edit. If you take the Apr 27, 2008 version of the article from Wikipedia and compare the two, you’ll notice just how closely the two mirror each other. The same can be done with Baidu Baike: a comparison of the differences between the Aug 29, 2006 Wikipedia article on Matteo Ricci and the Sept 6, 2006 Baidu Baike one. The overlap is stark and damning.

Again, this is not rigorous at all, merely a few cases that I have stumbled upon and require more rigorous examination before deciding whether the claims of plagiarism by Baidu and Hudong are unfounded, incidental (a few novice users unclear about citations), or systemic (either intentionally directed or through willful ignorance and refusal to correct it by those in charge).