Introduction

Bullipedia has its origin in elBulli (1962–2011), one of the most lauded restaurants of all time. Michelin awarded it with three stars (1976, 1990, and 1997), and more recently it was voted best restaurant in the world in 2002 and from 2006 to 2009 by industry authority Restaurant magazine (Williams 2012). The successful restaurant incorporated disciplines such as technology, science, philosophy, and the arts in its research, and Ferran Adrià, its owner and the most influential chef in the world, published his results in international conferences, books or journal articles, in a similar way to the academic process of peer review.

elBulli has now become elBulliFoundation, a center that seeks to be a hub for creativity and innovation in high cuisine, and continue the creation activity of the former restaurant. The key project of the foundation is an attempt to externalize all its wisdom onto the Bullipedia — Adrià’s vision for “an online database that will contain every piece of gastronomic knowledge ever gathered” (Williams 2012). He justified the need for such a culinary encyclopedia by claiming that “there is no a clear codification on cuisine” (Pantaleoni 2013). However, the Bullipedia is an idea yet to be developed. Thus, the question to answer at this point is: What should the Bullipedia be like? By analyzing different sources (specific literature, elBulli’s publications, interviews with Adrià, news, and emails with elBulliFoundation’s staff), we have identified several requirements that the Bullipedia must meet to achieve its mandate. We focus on encouraging user contribution and holding quality knowledge in this work.

For a project such as this (creating an online encyclopedia on cuisine) we believe that the collaboration of the community is indispensable to building quality contents. Bullipedia is an idea inherently 2.0 that can take advantage of crowdsourcing (Quinn and Bederson 2011; Doan, Dieter and Harmelen 2011) and harness the collective intelligence (O’Reilly 2005) to generate value. There exist many successful projects that could teach us valuable lessons for Bullipedia. The best example is Wikipedia, the online encyclopedia par excellence, whose success is due largely to its reliance on the crowd to create, edit, and curate its content. Other relevant cases are Allrecipes, Epicurious, and Cookpad, three of the most popular websites for recipe exchange. On these platforms, the members upload their own recipes and review and rate other members’ recipes.

A major concern in these kinds of projects that rely on their own users to succeed is precisely how to engage them. In our previous work (Jiménez-Mavillard and Suárez 2015), we recapitulated the main motivations for the crowd to create content. From them, in the context of this work, we highlight recognition and reputation as intrinsically rewarding factors that motivate the community to collaborate (Herzberg 2008). For this reason, we look into SO and we argue that its reputation system: 1) encourages participation, 2) is an excellent quality control mechanism that guarantees true value because it allows users to collectively decide whether or not contents are reliable, and 3) can be used to measure and predict the quality of these contents.

This paper is organized as follows. In section 2, we describe Q&A sites and in particular SO. In section 3, we outline the problem and significant related works. The experiment and the methodology is detailed in section 4, and the results are shown in section 5. We discuss the relevance of SO and other projects to Bullipedia in section 6, and finally, end with some conclusions and future work in section 7.

Stack Overflow and Q&A sites

Since the origin of the Internet, the volume of information has been increasing on the web, digital libraries and other media. Traditional search engines are helpful tools to tackle the abundance of information, but they just give ranked lists of documents that users must browse manually. In many cases, they simply want the exact answer to their questions, asked in natural language.

Q&A sites have emerged in the past few years as an enormous market to fulfill these information needs. They are online platforms where users share their knowledge and expertise on a variety of subjects. In essence, users of the Q&A community ask questions and other users answer them. These sites go beyond traditional keyword-based querying and retrieve information in more precise form than given by a list of documents. This fact has changed the way people search for information on the web. For instance, the abundance of information to which developers are exposed via social media is changing the way they collaborate, communicate, and learn (Vasilescu and Serebrenik 2013). Other authors have also investigated the interaction of developers with SO, and reported how this exchange of questions and answers is providing a valuable knowledge base that can be leveraged during software development (Treude, Barzilay and Storey 2011). These changes have produced a marked shift initially from mere websites born to provide useful answers to the question, towards large collaborative production and social computing web platforms aimed at crowdsourcing knowledge by allowing users to ask and answer questions. The end product of such community-driven knowledge creation process is of enduring value to a broad audience; it is a large repository of valuable knowledge that helps users to solve their problems effectively (Anderson et al. 2012).

The ever-increasing amount of Q&A sites has caused the number of questions answered on them to far exceed the number of questions answered by library reference services (Janes 2003), which until recently were one of the few institutional sources for such information. Library reference services have a long tradition of evaluation to establish the degree to which a service is meeting user needs. Such evaluation is no less critical for Q&A sites, and perhaps even more so, as these sites do not have a set of professional guidelines and ethics behind them, as library services do (Shah and Pomerantz 2010). Instead, most Q&A sites use collaborative voting mechanisms for users inside the community to evaluate and maintain high quality questions and answers (Tian et al. 2013). By a quality answer, we mean the one that satisfies the asker (Liu et al. 2008) and also other web users who will face similar problems in the future (Liu et al. 2011).

There are currently a variety of Q&A sites. The first Q&A site was the Korean Naver Knowledge iN, launched in 2002, while the first English-language one was Yahoo! Answers, launched in 2005. Some sites (such as Yahoo! Answers or Quora) are intended for general topics while others (such as Digital Humanities Q&A and SO) are formed by specialized communities centered on specific domains. In this paper, we are focusing on this last Q&A site.

SO is a popular Q&A site in which users ask and answer questions related to programming, web development, operating systems, and other technical topics. This forum was created by Jeff Atwood and Joel Spolsky in 2008, as an alternative to traditional sources of information for programmers, like books, blogs, or other existing Q&A sites. The fact that both Atwood and Spolsky were popular bloggers contributed to its success in the early stages of the project as they brought their two communities of readers to their new site and generated the critical mass that made it work (Atwood 2008; Spolsky 2008). This success was also promoted by its novel features; it incorporated a voting and Wikipedia-like edition system, and it was open and free, unlike earlier Q&A sites such as Experts-Exchange. Soon, the phenomenon of SO caught the attention of investors and media. The New York Times reported that the Q&A site raised $6 million in a first round of funding (Ha 2010), and additional rounds would raise further resources: $12 million in 2011, $10 million in 2013, and $40 million in 2015 (Stack Overflow 2016).

The site employs gamification to encourage its participants to contribute (Deterding 2012). Participation is rewarded by means of an elaborate reputation system that is set in motion through a rich set of actions. The main actions are asking and answering questions, but users also can vote up or down on the quality of other members’ contributions. The basic mode of viewing content is from the question page, which lists a single question along with all its answers and their respective votes. The vote score on an answer –the difference between the upvotes and downvotes it receives– determines the relative ordering in which it is displayed on the question page. When users vote, askers and answerers gain reputation for asking good questions and providing helpful answers, and they also obtain badges that give them more privileges on the website. In addition, at any point, an asker can select one of the posted answers as the accepted answer, suggesting that this is the most useful response. This also makes the asker and the answerer earn reputation. The reputation score can be seen as a measure of expertise and trust, which signifies how much the community trusts a user (Tian et al. 2013).

SO’s success is largely due to the engaged and active user community that collaboratively manages the site. This community is increasing both in size and in the amount of content it generates. According to the January 2014 SO data dump provided by the Stack Exchange network (and analyzed in this work), SO stores around 9.5 million questions, almost 16 million answers, and has a community with more than 4 million users. The number of questions added each month has been steadily growing since the inception of SO (Ponzanelli et al. 2014) and has reached peaks of more than 200,000 new questions per month (see Figure 1).

Figure 1

Number of questions by month (Ponzanelli et al. 2014).

The content is heavily curated by the community; for example, duplicate questions are quickly flagged as such and merged with existing questions, and posts considered to be unhelpful (unrelated answers, commentary on other answers, etc.) are removed. As a result of this self-regulation, and despite its size, content on SO tends to be of very high quality (Anderson et al. 2012).

Problem and related work

The primary idea of this study is to understand the relation between a question and its accepted answer in order to predict potential accepted answers (from a set of candidate answers) for new questions. We tackled this problem by analyzing SO to verify if the metrics associated to the reputation system activities – question’s score, answer’s score, user’s reputation, etc. – can decisively predict accepted answers for yet unresolved questions.

This problem has two main components: crowdsourcing and machine learning. The term crowdsourcing is the combination of the words crowd and outsourcing, and is the process of getting work from a large group of people, especially from online communities, rather than from employees or suppliers. This model of contribution has been applied to a wide range of disciplines, from bioinformatics (Khare et al. 2015) to the digital humanities (Carletti et al. 2013). “The Wisdom of Crowds” (Surowiecki 2005) is a popular science work for a general introduction to the concept. “Crowdsourcing Systems on the World-Wide Web” (Doan, Ramakrishanan and Halevy 2011) and “Human Computation: A Survey and Taxonomy of a Growing Field” (Quinn and Bederson 2011) give a scholarly broad review of the field. The former is focused on the Web as the natural environment for crowdsourcing. The latter emphasizes the power of humans to undertake tasks that computers cannot yet do effectively.

SO relies on the crowd, and some authors underlined the soundness of this user-generated content model to provide quality solutions. Vasilescu and Serebrenik (2013) investigated the correlation between the activity of SO’s users and their activity on GitHub, the largest social coding site on the Web. They demonstrated that the most productive programmers, in terms of amount and uniform distribution of their work, are the ones who answer more questions on the Q&A site. Therefore, a large number of answers on SO are presumably effective solutions, as they came from qualified programmers that incorporate good work practices.

Parnin, Treude and Grammel (2012) showed that companies like Google, Android, or Oracle also acknowledge the quality of contents produced on SO. These brands entrusted the documentation of their respective APIs – Google Web Toolkit, Android, and Java – to the SO community. The authors collected usage data using Google Code Search (currently shut down), and analyzed the coverage, quality, and dynamics of the SO documentation for these APIs. They found that the crowd is capable of generating a rich source of content with code examples and discussion that is more actively viewed and used than traditional API documentations.

The second pillar of this work is machine learning, a subfield of artificial intelligence that studies how to create algorithms that can learn and improve with experience. These algorithms learn from input observations, build a model that fits the observed data, and apply the model to new observations in order to make predictions on them. Machine learning is a transverse area used in a large number of disciplines, and with multiple applications, from computer vision to speech and handwriting recognition. “Machine Learning” (Mitchell 1997) is a classic introductory textbook on primary approaches to the field.

A recurrent problem solved with machine learning is classification. This is the problem of identifying the category of a new observation, and belongs to the type of supervised methods, that is, the task of building a model from categorized data. As the goal of our study is to determine when a question has been correctly answered, we posed this question as a classification problem: “Is this answer (likely to be) the accepted answer for this question?” The possible options are “yes” or “no.” Therefore, for every question each of its answers belongs to one of these two categories: “Yes” (it is the accepted answer) or “No” (it is not). The question seems trivial but even in many branches of pure mathematics, where specialists deal with objective and universal knowledge, it can be surprisingly hard to recognize when a question has, in fact, been answered (Wilf 1982).

While it is true that finding the “right” answer is ambitious, efforts to detect ‘good enough’ ones are underway. As already mentioned above, our approach is to extract features from questions, answers, and users, and apply classification to learn the relation between a question and its accepted answer. Many authors have applied machine learning techniques to provide the correct answer to a question by pursuing different objectives. For instance, some of them focused on directly identifying the best answer. Wang et al. understood questions and their answers on Yahoo! Answers as relational data (Wang et al. 2009). They assumed that answers are connected to their questions with various types of links that can be positive (indicating high-quality answers), negative (indicating incorrect answers) or user-generated spam. They proposed an analogical reasoning-based approach which measures the analogy between the new question-answer linkages and those of some previous relevant knowledge that contains only positive links. The answer that had the most analogous link to the supporting set was assumed to be the best answer. Shah and Pomerantz instead, evaluated and predicted the quality of answers, also on Yahoo! Answers (Shah and Pomerantz 2010). They extracted features from questions, answers, and the users that posted them, and trained different classifiers that were able to measure the quality of the answers. The answer with the highest quality was considered the best one. Interestingly, the authors reported that contextual information such as a user’s profile (they included information like the number of questions asked and the number of those questions resolved, for the asker; and the number of questions answered and the number of those answers chosen as the best answers, for the answerer) can be critical in evaluating and predicting content quality. This is actually a key finding in our experiment, as we will see in the results (section 5).

Another approach is to redirect the question to the best source of information. For example, Singh and Shadbolt matched questions on SO to the corresponding Wikipedia articles, so that the users could find out the answer by themselves (Singh and Shadbolt 2013). For this, they applied natural language processing tools to extract the keywords of the question, and then they matched them to the keywords of the pertinent Wikipedia article. Tian et al. addressed the problem of routing a question to the appropriate user who can answer it (Tian et al. 2013). They proposed an approach to predict the best answerer for a new question on SO. Their solution considered both the user’s interests (topics learned by applying topic modeling to previous questions answered by the user) and the user’s expertise (topics learned by leveraging the collaborative voting mechanism) relevant to the topics of the given question.

Similar to Q&A sites, we have Q&A systems. They are systems that receive a question in natural language and return small snippets of text that contain an answer to the question (Voorhees and Tice 1999). Related to these systems, Rodrigo et al. evaluated the correctness of answers as a classification problem (Rodrigo, Peñas and Verdejo. 2011). They studied two methods, F-score and ROC, and compared both evaluation approaches. Harabagiu and Hickl considered that the semantics of an answer is a logical entailment of the semantics of its question, and used this idea to enhance the accuracy of current open-domain automatic Q&A systems (Harabagiu and Hickl 2006).

Less relevant to our work, machine learning can be applied to classify questions instead of answers. Ponzanelli et al. presented an approach to automate the classification of questions on SO according to their quality (Ponzanelli et al. 2014). They investigated how to model and predict the quality of a question by considering both the contents of a post (from simple textual features to more complex readability metrics) and community-related aspects (for example, the popularity of a user in the community) as features. Other authors analyzed the problem of question classification by topic. Miao et al. compared two approaches, K-means and PLSA, to automatically classify questions on Yahoo! Answers, and reported the importance of incorporating user information to improve both methods (Miao et al. 2010). Blooma et al. also applied a machine learning solution but a different classifier, Support Vector Machine (SVM) (Blooma et al. 2008). Li and Roth developed a semantic hierarchical classifier guided by an ontology of answer types (Li and Roth 2006).

Results

Table 4 summarizes the performance of all the classifiers evaluated in our experiment. The results show that Linear SVC was the classifier with highest performance (for the four metrics) in comparison to the rest.

Table 5 displays the Linear SVC performance in detail. The evaluation was done on the testing set (7,285 quans). We can see that the classifier performed slightly better for “No” than for “Yes” (first and second rows respectively). The third row shows the average values for the total testing set of quans. Our model gave an accuracy of 88%, that is, it predicted correctly the accepted answer for 88% of the questions. This result is superior to others reported in similar works. Shah and Pomerantz (2010) measured the quality of answers on Yahoo! Answers with an accuracy of 84%, while Wang et al. (2009) identified the best answer for new questions, also on Yahoo! Answers, with a precision of 78%. Table 6 is the confusion matrix of the model. It represents the real number of “Yes” and “No”, and how Linear SVC classified them.

For visualization purposes, we also calculated the two most informative features (the two most helpful features for the classifier to classify the quans) and they were percent_answered_questions_q and percent_accepted_answers_a, i.e., the percentage of questions that were answered to this question’s asker (with respect to the total number of questions posted by this asker) and the percentage of answers that were accepted for this answer’s answerer (with respect to the total number of answers posted by this answerer), respectively. We trained a new classifier with only these two features and obtained an average accuracy of 82% (86% for “No” and 76% for “Yes”). This result is unsurprisingly inferior to the previous one (88%) as we left out other important features, but it demonstrated that the two most informative features alone provide enough information to the classifier for it to classify the quans with fairly high accuracy. These results are shown in Figure 2.

Figure 2

Space of quans: truth vs. prediction.

Figure 2 displays the space of quans created by the new limited model. Quans (dots) are drawn in a space that is divided into two physically differentiated areas: red for “Yes” and blue for “No.” The red area corresponds to high values for the two features – percent_answered_questions_q and percent_accepted_answers_a – and takes up approximately one third of the total space, while the blue area represents lower values for the features (the origin is the bottom-left corner). Analogously, every dot (quan) is either red (for “Yes”) or blue (for “No”). This color indicates the true category of the quan, whereas its position in the space means the category assigned by the classifier. The higher the values for the features, the higher their probability to fall into the red area. Thus, in an ideal scenario (accuracy of 100%), every quan would have fallen into its correct category (red dots and blue dots into red and blue areas respectively). However, the image shows that only 86% of blue dots fell into the blue area and, similarly, only 76% of red dots do it in the red area. This asymmetry was already observed in Table 5. Let us remember that this model only uses the two most informative features, and that the performance of the classifier improves (up to 88%) when we consider the whole set of features.

Finally, our experiment confirms Shah and Pomerantz’s (2010) results. They reported that contextual information such as a user’s profile (information like the number of questions asked and the number of those questions resolved, and the number of questions answered and the number of those answers chosen as the best answers) can be critical in evaluating and predicting content quality.

Discussion

The election of SO as a case study was not arbitrary. It could be argued that, against the backdrop of an online gastronomic encyclopedia, we should have looked into projects such as Wikipedia, Allrecipes, Epicurious, or Cookpad, that are, at a first glance, more relevant to Bullipedia than a programming forum like SO. Wikipedia is the most representative example of social knowledge on the Web. Its members self-organize and knowledge emerges from the bottom-up as a result of their interactions. It has a quality mechanism based on constant supervision by its own community, which ensures that contents are accurate at all times. On Wikipedia, articles get better and better over time as everybody contributes their knowledge, and therefore editions are less and less frequent as the content achieves a certain degree of consensus. This is what we call knowledge maturity: when the frequency of editions is lower than some threshold.

Despite its suitability, Wikipedia does not provide enough quantifiable data about contents and users. On one hand, the number of article views could be used as an indicator of the popularity of the topic but not for the quality of its content. On the other hand, user pages describe their remarkable activities and other recognitions, but their contribution for a particular article is difficult to measure. A good indicator for quality could be the life cycle of the contents in an article; however, this is not a trivial task. We would have to examine the revision history page, track its passages, calculate for how long every passage was or has been alive, and then create a measurement based on the passages that are currently active and belong to the article.

Allrecipes, Epicurious, and Cookpad are cooking websites where users create and publish their own recipes. Although these sites should have all the ingredients to be our model, they actually lack the social dimension needed for this study. Cookpad let the user favorite a recipe and comment on it, but there is no other consequence from this interaction beyond the favorite count for the recipe. Epicurious allows one to rate and write a review for a recipe but no other user can evaluate these reviews; therefore, there is no way to know how reliable they are. The same thing occurs on Allrecipes. Without a mechanism to evaluate the goodness of contributions, users’ feedback are just opinions – there is no evidence that they help to improve the original recipe.

SO is, however, a place where real problems meet their solutions. Although Q&A sites have a different nature from an online encyclopedia, there are some significant reasons that make SO relevant to the case of Bullipedia:

1. Topic. Food and computing seem to belong to distant worlds; however, there are important analogies between recipes and algorithms (Bultan 2012). Numerous researchers have proposed methods for converting a recipe text to a work flow by following different approaches – recipe feature extraction (Yamakata et al. 2013), machine learning (Mori et al. 2012), or domain constraints (Hamada et al. 2000).

2. Popularity. SO is one of the most popular Q&A based knowledge sharing communities on the web (Movshovitz-Attias et al. 2013), from which we can obtain valuable lessons for the Bullipedia.

3. Reputation system. A huge, active, and engaged community would be desirable to build up trust and quality content for the future Bullipedia, and a SO-like reputation system can help to fit this need. Recognition and reputation are intrinsically rewarding factors that motivate the community to collaborate (Herzberg 2008). Also, active users and reputed users encourage other users to contribute (Jiménez-Mavillard and Suárez 2015). Furthermore, building up trust is one of the major motivations for information exchange (Barachini 2009; Krogh, Roos and Kleine 1998). Thanks to this involvement, good questions and answers are easily identified by the community itself, and unhelpful posts are removed. This keeps the quality of content very high (Anderson et al. 2012). At the same time, the level of trust in a community and the value of the knowledge that it generates are also important factors in users’ willingness to collaborate (Krogh, Roos and Kleine 1998; Tsai 2000). All of this activity combines into a self-perpetuating cycle (Figure 3).

Figure 3

SO’s reputation system, contribution, quality, and trust.

The viability of Bullipedia depends on how we face some big challenges, namely, how to engage the community that will build the content, how to make this content quality knowledge, or how to compete with established recipe websites. As mentioned above, the reputation system fits the quality knowledge needs and the rewarding factors that motivate people to contribute; but social factors play a role equally decisive. Some of these are simply the common good or less altruistic motivators such as recognition and career advancement. Reputation is translated into expertise, which is very valuable for the development industry in the case of SO. Only time will tell if a similar reputation system on Bullipedia will be helpful in identifying the next generation of best chefs in the world. Bullipedia will likely also lure users away from other recipe websites, since Bullipedia has an important unfair advantage in its connection to elBulli and Ferran Adrià’s trademark. This is a project that has raised high expectations since its conception among culinary professionals and media. Moreover, Bullipedia is different from its main competitors because it will be more than a recipe exchange website – it will be a platform for recipe and knowledge exchange, and a forum for people to solve their questions, with a true social component incentivized with gamification.

Conclusions and future work

In this work, we have studied SO, a popular Q&A site for programming that owes its success to its committed community. This commitment is achieved by means of an elaborate reputation system and its triple role: 1) it is an implementation of the gamification employed by the site to encourage participation, 2) it is a collective mechanism to control the quality of the crowdsourced contents, and 3) it stores a reputation score for every user, that is, a measurement of their expertise on the site. These scores provide useful metrics to evaluate the quality of the contents. Thus, we contemplated using this reputation system to prognosticate the quality of the answers. In particular, we wondered whether it was possible to predict the likely accepted answer (from a set of candidate answers) for a yet unresolved question.

We formulated this issue as a machine learning classification problem: “for every quan (question-answer pair), is that answer (likely to be) the accepted answer for that question?” We gave two possible categories: “Yes” or “No”. In order to solve this problem, we first represented every quan as a feature vector; and then did a 10-fold cross-validation on our set of quans and trained and tested several classifiers with different parameters. The classifier that performed best was Linear SVC. Our key finding shows that this model correctly predicted the accepted answer for every question with an accuracy of 88%. This result is substantially higher than others reported in similar studies. Shah and Pomerantz (2010) measured the quality of answers on Yahoo! Answers with an accuracy of 84%, while Wang et al. (2009) identified the best answer for new questions, also on Yahoo! Answers, with a precision of 78%.

Our prediction is based on the thirteen extracted features for each quan, but the two most informative features were percent_answered_questions_q and percent_accepted_answers_a (the percentage of resolved questions posted by the asker and the percentage of accepted answers posted by the answerer, respectively). These two metrics suggest the importance of asking clear question in obtaining an answer and giving good answers in having them accepted. When a question is asked by a good asker and the answer is provided by a good answerer, the probability for the question and the answer to form a kuan (question-accepted_answer pair) increase rapidly (to 82% according to our second experiment and up to 88% according to our first one). These results reaffirm Shah and Pomerantz’s (2010) findings — they reported that users’ information is critical in evaluating and predicting content quality.

A question and its accepted answer constitute reliable knowledge, as it provides the solution for a specific problem that a user had in the past and that other users may face in the future. From this reliable knowledge (by identifying questions and their accepted answers) we can build a repository that contains exclusively quality knowledge on SO. This idea can be extrapolated to every possible domain. We have demonstrated that Linear SVC is suitable for Q&A classification, so, if we implemented a similar forum and reputation system on the future Bullipedia, it would be possible to apply this same idea to predict the best answer for unsolved questions on the gastronomic encyclopedia. Questions like “Is it safe to cook chicken in a slow-cooker?” or “What’s a good dressing for a salmon salad?” would have a best answer because our classifier would select it from all the supplied answers.

We believe that a Q&A forum with gamification implemented by a reputation system, plus elBulli and Adrià’s trademark, will form a delightful combination that would raise high expectations and engage a large enthusiastic community. This engagement would create quality knowledge and increase the potential of the Bullipedia project, from which the next generation of best chefs in the world might arise. In the future, we are planning to research if it is also feasible to use the reputation system to identify influential users in the community, like well-known chefs, other users with high reputations, good askers or good answerers, and improve the process of knowledge creation by analyzing the habits of these key actors.

Competing Interests

The authors have no competing interests to declare.