Gamification involves making into a game a ‘tedious’ task like writing an essay or washing the dishes. Jane McGonigal (2011), in Reality is Broken, describes a variety of situations where participants can be motivated by using gaming principles to enliven an otherwise tedious task. An example is Chore Wars (http://www.chorewars.com/), where you define the chores you don’t want to do and then get points for doing them as if the chores were quests in a fantasy role playing game. In her advocacy of gamification, McGonigal contrasts game life with “real” life: “Compared with games, reality is too easy. Games challenge us with voluntary obstacles and help us put our personal strengths to better use” (McGonigal 2011, 4). As the title to her book suggests, she believes that we can modulate the tedium and other problems, the “brokenness,” of everyday reality by thinking of it as a game and, where possible, designing game-like obstacles and rewards into it. This philosophy of gamification can be valuable to anyone trying to enhance motivation in learning. We take a different view: gamification involves providing feedback based on analytics of one’s activity. Whether you get a word count on how many words you have written at the bottom of your window in a Status Bar or get badges when you hit a target, in both cases you are getting feedback based on analytics.

We have built GWrit so that it can be a platform for analyzing and representing information about a user’s writing back to them in different forms, specifically in the form of:

Our working hypothesis is that users want information about what they are doing, and that gamification can be a playful way of representing that information back to the users so that they can make decisions and possibly be motivated differently. Gamification, for the purposes of this paper, is a process of re-presenting information in a different rhetorical mode. Instead of simply stating information about progress on a project (as in “you have finished 3 out of 5 tasks in 9 days”) gamification is an experiment in presenting this information playfully. GWrit is a platform into which we can plug and track a variety of analytics that gather information about writing; a variety of “serious” and “gamified” representations of that information; and tools for capturing information and comments about the writing we are trying to encourage through gamified analytics.

GWrit was initially designed to provide a place for users to define their writing tasks and overcome procrastination by getting points as they finish self-imposed tasks. In an early prototype we built a challenge mode where people could challenge each other to finish tasks.

With the opportunity to adapt it for university writing courses, GWrit was changed to provide an environment where instructors can design writing tasks with documented milestones for students to complete. The system keeps track of different types of behaviour, especially the completion of assignments, and uses rules to then assign badges based on activity (Figure 1).

The most important feature we built was a peer commenting system so that students could be asked to comment on each other’s writing. As will be demonstrated, this was heavily used (partly because its use was required in the course, but many students used it more than was required). Among the reasons for use beyond what was required may have been that the attention of others is a motivating form of feedback.

One last word on the gamification built into GWrit. The current limitation to the system as an experimental system is that the rules for assigning gamification have to be hard coded (Figure 2). One consequence of hard coding the badges is that to save time we limited the number of badges students could collect. Students collect the badges as they progress through the course (Figure 3). We have done the preliminary design work to imagine what an environment would look like where an instructor could define which activity leads to which badge.

One of the key digital humanities features of GWrit is analytics. The current version of GWrit has two analytical tools: Word Cloud and Concordance. In GWrit, Word Cloud provides a visual description of the written text based on word frequency; it gives greater prominence to words that appear more frequently in the writing text. For student users, it is an interesting form in which to visualize their writing and provides an easy way to have a visualization of the theme and vocabulary of the piece of writing. For instructors, this visualization helps to provide feedback effectively.

The concordance tool provides an interactive keyword-in-context (KWIC) visualization. Corpus and concordance resources for students have been described as one of the most promising ideas in computer-assisted language learning since the 1980s (Johns 1986; Leech and Candlin 1986) and have become key topics in language teaching. Concordance analysis in GWrit provides the context of a given word in the writing. For example, in the following Figure 4, the given word is “some,” and the right and left word context range is three words. The analysis shows the context of the word ‘some’, with three words before and after every appearance of the word “some.”

For student users, concordance analysis has the potential to reveal their usual grammar patterns, phrase patterns and sentence patterns. Making these patterns obvious may help some students self-correct their writing for some incorrect linguistic patterns.

With Google Analytics, we are able to assess the usability of the word cloud tool and other key events in GWrit. Figure 5 depicts submit comment events, checkout events, and word cloud events from January 2016 to April 2016, comprising a total of 16,865 events. A checkout event happens when the user checks out a writing task; a submit comment event happens when the user posts a comment on a piece of writing. The number of word cloud events followed the checkout event closely at the beginning of the winter term 2016. We can deduce that most users used the word cloud analytic tool after they checked out the tasks. Since users can edit their writing many times after checking out a task and they can also use word cloud on other people’s writing, the count of word cloud events exceeds the number of checkout events.

However, compared with the submit comment event, the number of word cloud events fell off rapidly from the initial assignment, and it was never picked up again by the majority of users. The reason might be that the submit comment event is tied to a grade, while the word cloud is not. The lack of activity could also be tied to users not seeing a value in the visual representation.

Cognitive Walkthrough (CW) is an evaluation method that aims to assess the usability of a system, and it focuses on ease of learning (Polson et al. 1992). This method is often used in the early stages of the design of a system and allows designers to anticipate some learning problems before implementation (Polson et al. 1992).

To conduct a CW test on GWrit, we assumed that GWrit users were new to GWrit and had experience with computers and the internet, but little experience with online writing systems. These four tasks were completed to evaluate the key features of GWrit:

Table 1 lists the correct actions sequence required to complete each task. Failure stories are presented to help identify the problems.

Task 2:

Failure story on action 2:

Criteria: Will the user try to achieve the right effect?

Problem noticed. There are two default tabs in the writing page: note and submission. Users may be confused about the difference between the two tabs and have no idea of where to write.

Task 3:

Failure story on action 3:

Criteria: Will the user associate the correct action with the effect he or she is trying to achieve?

Problem noticed. Users may easily find the analytic tools, including word cloud and concordance, but they may be confused about the goals of the analytic tools.

Task 4:

Failure story on action 1:

Criteria: Will the user know that the correct action is available?

Problem noticed. There is no obvious button or link to the submissions of other users, so users may find it hard to find others’ submissions.

Generally, CW evaluation identified both design and programming issues to be addressed to make GWrit a user-centred writing environment.

Heuristic analysis is a way of analyzing something based on a set of goals and measures of website usability. It is a way of auditing basic features of an interface. Through heuristic evaluation, the design team uncovered a few issues with usability. The heuristic used on GWrit was developed by Xerox for use on a variety of websites, and focuses primarily on user accessibility and website design (Pierotti 2018).

A major feature of GWrit is that a large number of conventions are borrowed from word processors, so the website as a whole is much more familiar to those who are used to using word processors. Users who do not have this experience are at a significant disadvantage when learning to use the website. The usability issues that are present are few but are of significant importance to the user. For example, users can lose their writing if they navigate away from a page without saving, and there is no integrated Help feature.

In terms of visibility, the system provides a good amount to the user at all times. Situations when the user is not totally aware of the status of the system are fairly uncommon. The interface is standardized for the most part, and there are relatively few different layouts for the user to learn and adapt to in order to understand the system. While the majority of features give users control and protect their work, it is possible to navigate away from the page without the user saving their work, causing all of the work since the last save to be lost. Other than that issue, the degree of user freedom is high. It is easy to change quickly between screens and tabs, access different menu levels, and change and edit information. Overall, GWrit is very effectively designed and accomplishes or exceeds the different goals of the heuristic review. Though there are some issues, as noted above, the website is user-friendly and should be familiar to even occasional computer users.

Figure 6 shows information from the GA dashboard. Since GWrit is only used at the University of Alberta and has not been opened to the public, only authorized users can log in to the GWrit system; therefore, most GWrit users are returning visitors.

The average session duration is 12 minutes and 12 seconds, meaning that most users spend about 12 minutes on the GWrit website at a time. This 12-minute time frame suggests that while users are not using GWrit as a composing environment (because, presumably, an assignment is written in increments of time longer than 12 minutes) they are also not using it only as an assignment submission system. They likely spend time reading or commenting on others’ work. It also suggests that the mutual study environment and commenting system, two major features designed to improve writing skills, are recognized by users.

Figure 7 shows the relationship between task deadline and the usage profile of GWrit. Writing Studies (WRS) 102 instituted different task deadlines in fall term 2015 (September 1, 2015–December 10, 2015) and winter term 2016 (January 1, 2016–April 10, 2016): in fall term 2015, assignments simply had a grading deadline; in winter term 2016, assignments had a draft deadline one week before the grading deadline and then a grading deadline. Figure 7 shows a comparison of the usage during these two semesters. The blue curve shows fall semester 2015, and the orange curve shows winter semester 2016. We tagged different task deadlines for WRS 102 in Figure 7. Points marked with a red circle are deadlines in fall semester 2015; points marked with a black circle are grading deadlines in winter semester 2016; points marked with a green circle are draft deadlines in winter semester 2016. By comparing the data from the two semesters, we can see in fall semester 2016 that the usage curve follows the deadline closely and reaches a peak at the deadline. In the winter semester 2016, the curve is more complex than in fall 2015.

There are two possible reasons for this:

To clarify the relationship between the deadline settings and usage profiles, we collected the data on assignment status including submit for review and submit for grade. In this way, we gathered the numbers of assignments submitted for review and submitted for grades from the different courses, and we extracted the data for WRS 102. In Figures 8 and 9, the blue line shows the curve of assignments submitted for review, and the orange line shows the curve of assignments submitted for grade. We can clearly see the difference between the curves during these two semesters: the assignments submitted for review reach a peak about one week before assignments were to be submitted for a grade in winter semester 2016. In fall semester 2015, the shape of the blue and orange curves is similar: they reach a peak on the same date.

In addition, we found a relationship between the user profile activity and the 3-week module schedule for WRS 102. In winter semester 2016, after the first week, WRS 102 students spent three weeks each on Module 2, Module 3, Module 4, and Module 5. After comparing the curve and the time duration of each module, we found four peaks that follow the four-module duration closely, and the assignments submitted reached a peak at the end of each module.

In general, this comparison of the usage profiles in the two semesters demonstrates that the GWrit usage profile largely depends on the deadline settings and the study module settings. In other words, GWrit is a deadline- and module-driven study environment in its current usage.

GA provides a report on how users access GWrit and the device and operating system preferences. From these reports, we have the following findings. Most users (79.23%) visit GWrit through a referral website, and the most used referral website is accounts.google.com. Users use this referral because their email through University of Alberta requires them to login through accounts.google.com to confirm their login authorization. The second source is through a search engine, and the most common search keywords are “gwrit,” “game of writing,” “gwrit ualberta,” and the url of the GWrit webpage (see Figure 10).

Chrome, Safari and Firefox are the top three browsers used, vastly exceeding the alternatives (see Figure 11). Macintosh and Windows are the top two operating system with a share of almost 90%. However, iOS and Android systems together take 6.2% of the share. Future development needs to focus on the compatibility of GWrit with the most commonly used browsers and operating systems (see Figure 12).

Through analyzing the session numbers and averaging the session duration of the three devices commonly used (desktop, mobile, and tablet), we find that most sessions happened on a desktop (see Figures 13 and 14). But after comparing the average duration of the sessions using the three devices, we find that the average duration on the three devices was similar, but even longer when users chose a mobile phone or tablet to visit GWrit. However, based on the development cost and the session length on tablets, we have decided against developing a version for tablet users at this stage.

GA also provides a report on how users engage with the website. Figure 15 presents a behaviour flow report that visualizes the path that users traveled from one page or Event to the next. This shows the traffic route after users had logged in. We generated a flow chart (Figure 16) that illustrates the route straightforwardly. This data shows that, after logging in, most users go to the course panel, which lists all the projects in that course. Clicking a course name leads users to the project panel, where tasks are listed. Then the traffic flow divides into two branches, “task submitted” and “task panel.” Users follow the first branch to view others’ submissions, where they can post or reply to comments about the draft. Users following the second branch usually finish their own task and use interactive tools like word cloud and post comment.

In addition to the high traffic route that shows the most commonly used pathways in GWrit, we also found that 7% of users went directly to the “task submitted” panel after logging in, where they viewed others’ submissions, selected one to view, and posted comments or replied to comments. This pathway might be explained by a second user group of teaching assistants and paid peer tutors who usually go directly to the submission list to view and comment on students’ posted assignments. This separate user group, thus, constituted most of the 7% who went directly to the submitted panel. However, it is not possible to separate data generated by these different user groups, so we would need to further analyze user behaviour in these different groups by tagging different users on GA.

GWrit incorporates three task completion structures to help students. One structure, the course completion fuel gauge, allows students to track their progress through the course and provides visual feedback as to the relationship between work completed and work outstanding. The second structure, a task list, describes a process to follow to complete the selected assignment. The third structure, assignment deadlines, gives students due dates to submit drafts of the assignments. All three of these task completion structures were used in WRS 102 in winter 2016 term and the first and the last structures were used in fall 2015 term. In a notable exception to this pattern, in the fall term 2015, students were told that the deadlines for drafts of assignments (one week before the marking due date) were optional. Consequently, many students did not post drafts of their work in time to get formative feedback.

Of the three task completion structures available for use in GWrit, the assignment deadlines seem to have been most effective in motivating students to complete their work in the course. This is hardly surprising, given the reward structure of education and the mindset of contemporary post-secondary students. While they may have been entertained by the changing colour of the fuel gauge over the term as they completed more and more work, it did not seem to drive them onward. The task lists may have been largely ignored as activities because they did not clearly link to course-related rewards. The Game of Writing as it is used in the academic writing course constitutes a serious game where students learn how to excel by becoming proficient writers of academic prose. Their decisions and choices are driven by time-constraints and whether or not an activity will help them reach their goals. When deadlines are tied to marks, students make time to submit a draft to receive feedback; when they receive and act upon useful feedback, they are able to improve the quality of their assignment drafts.

The qualitative improvement in the quality of student ideas and expression over the term suggests that the commenting function in GWrit assists students in learning the core principles of the course and exhibiting those in practice. In a survey of students over four years in the academic writing course that used GWrit, we asked the students to rate the following statement on a scale of 1–5 (with 5 = strongly agree): “This course has improved my understanding of key concepts about academic writing” (see Table 2). This data shows that students feel they improve their ability to write in academic contexts, and it suggests that the use of the gamification system to teach academic writing had positive results.