Research Design

This research project has been designed as a case study of a particular group of individuals within a specific institutional context. It involves a mixed method approach to data collection which includes both quantitative and qualitative data. The outcomes of this research are descriptive and hope to provide the basis for designing subsequent further research.

Sampling considerations

This study is concerned with aggregates of properties and tendencies relating to first year undergraduate students who commenced their studies at Middlesex in September 2007 but who have decided to temporarily interrupt or completely withdraw from their courses. This study specifically looked at students who had taken the decision and had informed the University of their intent by the end of November 2007. This has been done for practical reasons, however, it is consistent with the literature which states that students tend to withdraw from their courses within the first 6-8 weeks of commencing them (Parmar and Trotter, 2004).

Secondary data was also retrieved from the university’s student management system, MISIS, with regards to students’ age, sex, their programme and their date of withdrawal. The sampling strategy went through a number of iterations ranging from looking at sampling modules with varying levels of online activity, to sampling programmes which are historically prone to high attrition rates, to taking a purposive sample based on the reasons for withdrawal whilst taking account of age and sex. An extract of the research diary illustrates how the strategy changed on almost a daily basis (appendix 1). However, after numerous meetings and emails with the Academic Registry (AR) exploring what information is held on students, it appeared that by the end of November 2007 only 98 students had a withdrawn or interrupted status code in MISIS who fully met the specified criteria. This made it possible to look at the full number of possible participants thus avoiding complex sampling designs and making allowances for differing probabilities in the selection.

Once a full list of research participants was established letters were sent to both term-time and home addresses informing them of the research project and offering them the opportunity to opt out. Of these, 4 ex-students chose to opt out and 1 letter was returned as undeliverable. Also excluded from the research was 1 student, who appeared to have dropped out this year, but has been attempting to start a course for a number of years and has postponed it twice in the past. Such a case could provide an interesting life history approach to HE studies which could form the basis for undertaking subsequent research on HE entrants, but not one which would help answer the research questions. Thus the full number of participants was reduced to a manageable 92 ex-students.

Data Collection

The first part of the research project deals with ‘reality’ in a quantitative manner. It deals with properties of individuals’ interactions and their values which can vary and can be compared, essentially translating these into variables.

Carrying out this research relied on the use of the built-in tracking tools of VLEs and specifically those of the Blackboard Learning System (Vista). The VLE records interactions of individual users and stamps them with a date and time. These records are accessible to both staff and students, and provide an accurate record of online interactions which is meant to be used as a monitoring tool by staff and as a study management tool by students (where they’ve been, how much time they spent online, etc). The level of detail recorded by such systems is significant and can be accessed per module, per student, per session, etc.

Using tracking data immediately questions how the researcher views the VLE and brings into question how they approach the research. As all interactions are date and time stamped the use of tracking tools can be likened to systematic classroom observations, albeit online, which are closely linked to ethnographic approaches to research. The VLE becomes the ‘complete observer’ (LeCompte and Preissle, 1993 cited in Cohen, Manion and Morrison, 2000, p. 310) which records all interactions as and when they happen (event-driven rather than time-driven), allowing researchers to tap into many of the advantages that systematic classroom observation affords. This includes the gathering of large data sets which afford generalisable findings, high validity for observable behaviours (although mental activities are not accounted for) which are precisely recorded and the ability to reduce and filter highly complex phenomena into manageable data. A level of participant bias may be introduced in this type of observation by computer savvy students who are aware of how their tracking information might be used. There may also be cases where students are given extra credit/marks for participating online on a regular basis.

However, as the information being collected by the system is not done for the purposes of this research study, it is possible for the researcher to treat this information as secondary data. The VLE can not tell researchers from where students were accessing their online course, what environmental conditions were present, if they were alone whether they had others assisting them in their online tasks, or whether they were interacting with their online course as a primary task or whether they were using it in the background for referencing purposes. This method does not allow for the researcher to be fully immersed in the day to day lives of the participants as is necessary in an ethnographic approach (Robson, 2002). It was felt that this information was crucial in understanding the data if it was to be examined as systematic observational data and as such it was decided that the information collected by the VLE would be treated as secondary data due to the lack of information with regards to context. The level of observation afforded by the VLE was not enough to make generalisations about this particular group of students. It was envisaged that the methodological triangulation would be able to shed light on the context and why interactions occurred in this particular manner, lead to the understanding of the level of participant bias and substantiate any findings.

In the first part of this research the software and its developers in this instance are acting as the standardised research instrument. By using the VLE tracking data a level of observer bias is introduced by the software manufacturers rather than the researcher. Tracking tools are influenced in their design by the programmers’ personal constructs, theories of education and experiences of achievement and attendance monitoring. Software programmers choose to record ‘behavioural by-products’, the physical effect of interactions which remain after the events have taken place (Barlow et al., 1984 cited in Cohen, Manion and Morrison, 2000, p. 347) known as ’trace measures’ based on layers of theoretical underpinnings which are not evident. Hakim (1987, cited in Cohen, Manion and Morrison, 2000, p. 362) discusses the design of research projects based on secondary data and claims that they are studies that are designed ‘back to front’. Instead of purposely deciding what information to collect in the research design stage, the researcher finds out what is already being collected and then designs the research around it. In the data analysis phase of the project the variables to be analysed are a priori known, thus making the coding frame much simpler to design than in other research projects.

Beaudoin (2002), however, cautions against solely relying on this type tracking facilities offered by VLEs by noting that many students who fail to actively participate in a face-to-face or online class still achieve the intended learning outcomes and do academically well despite their apparent lack of interaction. ‘Lurking’, defined as logging in, observing but not contributing to online forums, although not a highly visible form of learning, is still a legitimate method of learning. Pappas, Lederman and Broadbent (2001) state that tutors need to rethink the way they monitor student performance due to the lack of visual and aural feedback in an online environment. However, in their study ‘lurking’ is not recognised as a preferred method of learning in any of the three cases they report on. The limitation of tracking tools is also highlighted by Hewling (2004) who examined the effectiveness of these tools with regards to students who lurk as well as those with limited access to the internet, who prefer to log in once, download materials and engage with them offline, even though they are formally enrolled on an online course. Consideration of this point is important for the construct validity and reliability of the study.

As no position or method will provide and undisputable clear view of the empirical field (Brown and Dowling, 1998) methodological triangulation has been sought through the use of a mixed method approach. Once the analysis of the secondary data was completed it was envisaged that a short description of each participant’s interactions would be mailed to them as a means of member checking (sample profile in appendix 2). As the participants are no longer studying at the University and may be physically remote, short semi-structured telephone interviews would follow in order to corroborate the findings of the secondary data analysis.

The interview schedule (appendix 3) was designed based on the limitations and difficulties encountered in the first phase of data collection and subsequent analysis and on the need to confirm any assumptions held. The questions fell broadly under five main categories:
  • · Self-identity - which took account of the theoretical perspective of symbolic interactionism and positionality theory
  • · Level of participant bias – which aimed to collect information with regards to external factors and influences
  • · Clarification of participants’ intentions – which took account of the theoretical concepts of learning held by the researcher
  • · Context – which took account of the theoretical concepts of space held by the researcher and aimed to address the shortcomings of tracking data discussed above
  • · Other – which allowed for participants to make any statements or ask questions.
The revision and updating of this instrument is also discussed in the next section of this paper as data collection is still ongoing.

Data analysis

A coding frame was developed based on the information provided from the VLE which dictated the number of variables to be considered. Each piece of datum gathered from the VLE was ascribed a code (coding book in appendix 4) and entered into SPSS for analysis. The variables concerned specifically looked at whether students logged on, how often, what time of day, what type of materials and tools they accessed as well as personal detail with regards to age, sex and the subject to which they were aligned whilst studying at Middlesex University. With regards to the type of materials and tools accessed the coding frame considered the the structure of VLEs as specified by Anagnostopoulou, Haynes, Bakry and Jackson (2003). VLEs consist of four main categories of tools: content delivery tools, assessment tools, communication and collaboration tools and management tools and as such all items accessed by students were coded appropriately.

Through ‘progressive focussing’ which according to Pralett and Hamilton (1976, cited in Cohen, Manion and Morrison, 2000, pp. 147-148) means moving from the wide angle view to the more specific salient points of the emergent picture a number of interesting observations were made which are discussed in the relevant section of this report. The collected quantitative data provided by the VLE was then qualitized, thus transforming it into qualitatively describable profiles of each student’s online interactions (appendix 2) which also forms part of the next phase of data collection. Speculative explanations for the emerging picture’s key elements and possibly their causes were then documented thus commencing the process of hypothesis generation.

Inferences made during this phase of analysis led to the updating of the design of the interview schedule. Key issues and areas for subsequent investigation were added appropriately. Having ‘finalised’ the interview schedule, feedback was sought from two researchers who are familiar with the project and their comments taken onboard leading to yet a further revision of the schedule. It is anticipated that interviews will take place at in the next month, which will hopefully allow enough time for the responses to be analysed and reported in the final version of this paper. Without the interview data it is unlikely that discussion of findings can move on from hypothesis generation to theory generation.