Data-Driven Usability Assessment:
Using Multiple Forms of Online Feedback to Improve E-Learning Design
Melissa E. Pierson
Department of Curriculum and Instruction,
Sara G. McNeil
Department of Curriculum and Instruction,
Bernard R. Robin
Department of Curriculum and Instruction,
Abstract: Data drives educational decisions that allow
instructors to immediately respond to learner needs. This paper will
demonstrate how the use of online tools, both transparent and user-initiated,
can yield data to guide not only the maintenance of learning sites, but also
the customization of e-learning environments to more precisely meet the site
use and learning needs of learners. Focus will be on introducing data
collecting tools and models; strategies for analyzing data and implementing
data-driven changes exemplified through case studies, and implications of data
for designing and maintaining e-learning environments.
Introduction
The days of placing course materials online and crossing our fingers in the hope that the online learner’s needs will be met are over. Online educational environments should be considered live spaces that can and should be responsive to the needs of each particular learning audience.
Data drives educational decisions that allow instructors to immediately respond to learner needs. Whereas instructors in a face-to-face learning context often have the added advantage of quizzical looks and informal questions that can signal a need for redirected instruction, effectiveness of online course design is often not known until students are formally assessed. In fact, e-learners might be quite adept at hiding misconceptions or struggles given the increased amount of time they have to hunt around a website looking for the right information or to post contributions in an asynchronous environment. An instructor might never know the extent to which their online learning aides are meaningful or counter-intuitive, or even how often students attempt to access those materials.
Online course designers can use feedback to improve teaching by monitoring student progress, controlling the pace of learning, and evaluating teaching strategies (Hazari & Schno, 1999). Both user-initiated and more behind-the-scenes feedback allows the instructor to modify e-learning materials on an ongoing basis. Armed with this type of data, faculty are continuously able to monitor students' learning throughout the e-learning experience.
The collection of data from website users can aide “usability engineering,” which attempts to create more usable Web sites by putting the users’ needs at the forefront (Instone, 1997). Understanding what e-learners need and intend to accomplish is becoming of primary concern to e-learning designers and instructors. Ongoing data-driven needs assessment can benefit both formal educational websites such as course web pages in higher education to more informal online learning tools, such as community learning sites. These strategies are also of use for both materials created with course creation tools and custom user-developed websites.
This paper will demonstrate how the use of online tools, both transparent and user-initiated, can yield data to guide not only the maintenance of learning sites, but also the customization of e-learning environments to more precisely meet the site use and learning needs of learners. Specifically addressed will be the use of web-use statistics software that operates behind the scenes to collect site use feedback virtually unbeknownst to students, and online forms and comment fields that allow site creators to interact with students directly, soliciting the feedback that students choose to share. Conference session attendees will specifically
· be introduced to data collecting tools and models;
· learn strategies for analyzing data and implementing data-driven changes, as exemplified through through case studies; and
· engage in a discussion of other forms of feedback and how those formats might be used to enhance the e-learning experience.
Transparent Data Collection: How it Works
Web-use statistical software, or log analyzers, can be a powerful data collection tool for e-learning developers and instructors. The strongest, most completely sound instructional design might be proven useless if the learning materials and structure do not match changes with student needs. Examples of types of data that can be collected in one of these statistical tools, LiveStats, along with ways strategies for data analysis and implementation are listed below:
· Download: A hit or request for an element considered to be a downloadable file. Online courses frequently offer downloadable documents for students to use or print. Time must be taken to ensure consistent, accessible formatting. Awareness of how often documents of this type are accessed will give instructors an idea of how vital they are to students.
· Hit: A hit is any request made on the web server. This includes page views, requests for images and requests for downloadable files. Total site hits can provide overall usage data to support the case for additional course materials or can give an indication of pages that are being overlooked.
· Page View Errors: A hit or request for an element considered to be a page element that was not successful. Knowing that certain elements of an instructional site are not functioning properly can give instructors valuable information so that corrections can be made even before students attempt to access the material.
·
Visit:
A group of transactions between an IP
address and the web server. Details on the actual visits to the site can be
important indicators of student interaction with site materials. This type of
data can answer, for example, the question of when students access the site. If
most students are working from home at
· Time Spent: Number of seconds between the first and last request of a visit. The time a user spends on a site can be an indication of how long it takes to access and read material.
· Visitor: Someone who has initiated a 'Visit' on the web site. Instructors can see who, in fact, is using the course materials. Depending on the intent and purpose of the e-learning environment, it is possible that such data can lead to decisions about protecting course elements from non-students or making other materials available in alternate formats or languages.
· Browser Tag and Operating System: Two commands that identify when a web browser makes a request, the type of browser and operating system the user is using. Online course developers make assumptions about the hardware and software that users have. The usability of a site depends greatly on how compatible the information is with an individual user’s computer set-up. If it appears that a great number of users have a browser or operating system that are less or more functional than initially determined, modifications in material format can be made.
· Viewed Once: If a page is the only page viewed in a visit, it is a 'Viewed Once' page. Depending on the intent of an e-learning site, it is likely that developers and instructors expect students to access more than one page. If pages are viewed once, it may be that the viewers are non-students, and thus the site may be too easy to access for non-students.
Transparent Data Collection: Cases in Action
Case 1: The Grandeur of Viceregal
Figure 1: The Grandeur of
Viceregal
The Grandeur of Viceregal Mexico, Treasures from the Museo
Franz Mayer website was a collaborative effort between the
The following usage
data for the site were collected using two different versions of the program, LiveStats. The collected data show the number of web
visitors that viewed the Grandeur site and their patterns of use for the period
that the exhibition was on display at the museum.
Total Number of Visits/Time Spent Viewing Site
During the
exhibition time period,, there were 18,804 unique user sessions, or visits to the Grandeur of Viceregal
Mexico web site. The smallest number of visits occurred on a Sunday,
with 49 visits, and the largest number of visits occurred on a Wednesday, with
602 visits. On average the smallest number of visits occurred on a Saturdays
with 209.44 visits. On average the largest number of visits occurred on a Mondays
with 314.28 visits.
32,634 pages from
the Grandeur site were viewed during this period, and the average time
spent per visitor during the selected range was 1 minute and 57 seconds.
Most Commonly Viewed Pages
As one might expect,
the main introductory page (default.hm) was the most commonly accessed page of
the Grandeur of Viceregal Mexico web site. As shown in Table 1, the data showed
that other frequently accessed pages included the cultural exchange,
exhibition, and lesson plan pages.
|
Top
10 most commonly accessed pages during period: |
||||
|
/default.htm (main
intro page) |
with |
2482 |
page views |
(7.65% of all traffic) |
|
/exchange/visit_2.cfm |
with |
1302 |
page views |
(4.01% of all traffic) |
|
/exhibition.html
|
with |
736 |
page views |
(2.27% of all traffic) |
|
/resources/lessonplans.htm
|
with |
730 |
page views |
(2.25% of all traffic) |
|
/exchange/exhibit.cfm |
with |
725 |
page views |
(2.23% of all traffic) |
|
/exchange/meet.cfm
|
with |
595 |
page views |
(1.83% of all traffic) |
|
/comparison.html
|
with |
437 |
page views |
(1.35% of all traffic) |
|
/timeline/1500s.html |
with |
434 |
page views |
(1.34% of all traffic) |
|
/resources.html
|
with |
417 |
page views |
(1.28% of all traffic) |
|
/exhibition/cultures.html |
with |
411 |
page views |
(1.27% of all traffic) |
Table 1: Most Commonly Accessed Pages
Where Visitors Came
From
Table 2 shows that visitors to the Grandeur web site came
from across the globe. The majority of Web visitors were from
|
Breakdown of Traffic by Continent |
|||
|
|
with |
15120 sessions. |
(80.93 % of traffic) |
|
|
with |
149 sessions. |
(0.80 % of traffic) |
|
|
with |
955 sessions. |
(5.11 % of traffic) |
|
|
with |
329 sessions. |
(1.76 % of traffic) |
|
|
with |
208 sessions. |
(1.11 % of traffic) |
|
|
with |
28 sessions. |
(0.15 % of traffic) |
Table 2: Breakdown of Visitors Worldwide
Search Engines
Information about how visitors to the Grandeur web site used search engines also provides some insight into how web visitors search for information online. An overview of search terms that visitors entered using the popular search engine, Google, is shown in Table 3. Of interest to educators is that the two most frequent search terms were “lessons,” and “plans.”
