Signaling Does Not Adequately Improve Diary ComplianceJoan E. Broderick, Ph.D. and Joseph E. Schwartz, Ph.D.
Department of Psychiatry & Behavioral ScienceStony Brook University
Saul Shiffman, Ph.D.University of Pittsburgh, and
invivodata, inc., Pittsburgh, PA
Michael R. Hufford, Ph.D.invivodata, inc., Pittsburgh, PA
Arthur A. Stone, Ph.D.Department of Psychiatry & Behavioral Science
Stony Brook University
ABSTRACT
Hypothesis: Compliance with a paper diary protocolwould be improved by using auditory signaling. Background:Prior research has demonstrated that compliance with the re-porting schedule in paper diary protocols is poor. Methods:Adults with chronic pain (N = 27) were recruited from the com-munity to participate in a 24-day experience sampling protocolof 3 pain assessments per day (10:00 a.m., 4:00 p.m., 8:00 p.m.).Diaries were instrumented to record openings and closings,thereby permitting determination of date and time when the par-ticipant could have made diary entries. Participants were sig-naled with a programmed wristwatch at the onset of each30-min assessment window. Two compliance windows were de-fined: � 15 min and � 45 min of the targeted assessment time.Results: Self-reported compliance based on participants’paperdiaries was 85% and 91% for the 30- and 90-min windows. Ver-ified compliance was 29% and 39% for the two windows. Sig-naling produced a significant increment in verified compliancewhen compared with an identical trial without signaling. A sig-nificant eroding of verified compliance was observed across the3 weeks of the study. Conclusions: Self-report dating of diaryentries may be misleading investigators about compliance withdiary protocols. Although auditory signaling enhances compli-ance, the result is still unsatisfactory.
(Ann Behav Med 2003, 26(2):139–148)
INTRODUCTION
For decades, researchers and clinicians have used diariesto capture patients’ experiences, such as pain, mood, and med-ication use in the natural environment, often in the context oftreatment trials (1). The rationale for the use of diaries is basedon an extensive literature documenting the inaccuracies andbiases that enter into retrospective reports (2–6). Research inautobiographical memory has shown that the past is recon-structed and is influenced by salience of past events (2), moodat time of recall (7,8), personality (9,10), and current level ofthe experience or symptom (11,12). Bradburn (13) reported asystematic bias in recall of events such that respondents over-estimate the frequency of events when asked to recall over along time frame and underestimate events over a short timeframe.
Recognition of these recall biases has led to widespreadadoption of diary methodologies to collect patient experiencedata close to when it happens. Collection of patient experienceon a daily or even more frequent basis (several times a day) isviewed as superior to assessment methods that require the pa-tient to recall symptoms over days or weeks (14). Assessment ofexperiences in the natural environment also yields data that haveecological validity and can capture associations between spon-taneous events and responses that the patient may not recognizeand not report retrospectively (15). This approach to assessmenthas come to be known as the experience sampling method (16)or ecological momentary assessment (17). These methods havebeen instrumental in improving the reliability and validity of pa-tient experience assessment (18).
The specific vehicles for capturing in vivo experiences in-clude paper diaries and electronic devices, such as handheldcomputers. The typical format is for the patient to provide rat-ings at times determined by the researcher or clinician, thusgaining a view of the patient’s experience during his or her rou-tine activities. Hundreds of studies have used diaries to gatherdata on a wide range of experiences, such as intrusive thoughts,episodes of cardiac pain or asthma, voiding of urine, pain, medi-cation dosing, cigarette smoking, and food intake. The timing of
Disclosure: invivodata, inc. supplies software for the use of electronicdiaries in clinical trials. Saul Shiffman is a co-founder and the ChiefScience Officer of invivodata, inc. Michael Hufford is Vice Presidentfor Scientific Affairs at invivodata, inc. Arthur A. Stone is AssociateChair of the Scientific Advisory Board of invivodata, inc.
This research was supported by National Cancer Institute GrantCA–85819 (Arthur A. Stone, Principal Investigator) and in-kind sup-port from invivodata, inc., Pittsburgh, PA.
We thank Daniel Arnold, Doyle Carney, Julie Grassell, Corey Harmon,and Jill Jackowski for assistance with completing this study.
Reprint Address: J. E. Broderick, Ph.D., Department of Psychiatry &Behavioral Science, Putnam Hall, Stony Brook University, StonyBrook, NY 11794–8790. E-mail: [email protected]
the assessments can be fixed (same time[s] each day) or random(a predetermined density of sampling times randomly distrib-uted across the waking day) (16). End-of-day diaries are verycommon; they are minimally burdensome to patients, do not re-quire expensive technology to implement, and limit recall biasby reducing the length of recall to a day.
Paper diaries, the focus of this article, are by far the mostcommon form of diaries seen in the literature. Patients are in-structed in how to use the paper diary, to complete it on the pre-determined sampling schedule, and to record the date and timeof each entry (19–22). Sometimes researchers telephone the pa-tient periodically to inquire about compliance with diary com-pletion, but often patients are expected to complete the task ontheir own for several weeks at a time (23–25).
Some researchers using repeated, within-day sampling pro-tocols (e.g. 10:00 a.m., 12:00 p.m., 4:00 p.m.) have signaledpeople to remind them when it was time to make a diary entry.Furthermore, signaling was required to implement more com-plex, random sampling schedules. Radio-triggered pagers orprogrammed alarms on watches were introduced to prompt pa-tients to make diary entries (26–35). In these studies, patientswere told to wear the signaling device and to carry the diary withthem so that an entry could be made whenever they were sig-naled. In these protocols, patients typically are informed aboutthe rationale and importance of compliance with the samplingprotocol, are urged to make every effort to adhere, and are askedto report honestly the time and date of each of their en-tries—even if not consistent with the protocol.
Diary researchers, ourselves included, have generally as-sumed that participants can and will follow a sampling protocol.Self-reported dates and times of entries are assumed to be accu-rate, and compliance with the diary is reported as the number ofcompleted diaries returned to the researcher—typically in therange of 80% to 90% (36–38). For example, de Wit and col-leagues reported 100% compliance in over 50% of cancer pa-tients making pain intensity ratings in a paper diary twice perday for 8 weeks (23). On the basis of these self-dated diaries, itappeared that diary compliance was adequate.
Examination of the compliance literature more broadlyraises concerns about the veracity of patients’ self-reportedcompliance. Patient compliance with medication dosing hasbeen found to be a significant problem, despite patients’ reportsof adherence (39–42). The terms parking lot compliance anddumping have been coined to describe the practice of dispensingmedication just prior to a research or clinical visit so that the pa-tient appears to have used the prescribed amount of medicationsince the last visit (43). Self-report of medication use and datafrom dosing monitors can be highly discrepant, with self-reportsoverestimating medication use by 50% or more (44). Likewise,self-report in diaries has been shown to be highly discrepantwith the data collected on monitors for glucose levels (45) andblood pressure (46). Compliance with a salivary cortisol sam-pling schedule in a 1-day research protocol was significantlyless than self-reported compliance, and the samples collected ina noncompliant fashion produced results that were significantlydifferent from those that were compliant (47).
With regard to symptom diary completion, a recent studyconducted in our laboratory explicitly compared compliancerates calculated on the basis of patients’ reports of when their pa-per diary entries were completed (recorded on each diary entryat the time of completion) with electronically verified compli-ance rates (48). In that study, chronic pain patients were in-structed to complete pain diaries three times a day over a 24-dayperiod at the same times every day (10:00 a.m., 4:00 p.m., 8:00p.m.). Auditory signals reminding participants to complete dia-ries at the specified times were not used in the paper diary condi-tion. The reported compliance rate (defined as being madewithin ± 15 min of the specified time) based on patient entries ofwhen diaries were completed was 90%. However, the actualcompliance rate was only 11% when verified with electronicmonitoring of the openings and closings of the diary. In the sec-ond arm of that study, an electronic diary with features designedto enhance compliance, such as signaling patients when theywere to make diary entries and providing weekly feedbackabout their actual compliance, yielded a verified compliancerate of 94%, thus demonstrating that good compliance could beachieved in this patient population.
This study was designed to determine whether the use of asignaling procedure for paper diary data collection is an effec-tive method to enhance diary compliance. We used a newly de-veloped instrument that electronically records the time anddate of when the diary is opened and closed, the same deviceused by Stone et al. (48). This allowed for a comparison of pa-tients’ self-reported completion of diary entries with the dateand time of openings and closings recorded by the instrumen-tation. We hypothesized that use of a prompting signal wouldresult in significantly improved compliance compared with therate observed for the unsignaled diary in our prior study,which was conducted with the same population and paperdiary methodology (48). We also hypothesized that signal-ing would minimize the degradation of compliance over thecourse of the study.
METHOD
Participants
Recruitment for study participants was done through radioand newspaper advertisements and fliers in hospital and outpa-tient clinic areas in an academic health sciences center. Adults(ages 18–70) who experienced chronic pain were invited to in-quire about the study. Each adult was screened on the telephoneto determine if the following eligibility criteria were met: (a)pain on 3 or more days per week, (b) at least 3 hr of pain per day,(c) rating their “typical pain” as 4 or greater on a 0 to 10-pointpain severity scale, (d) negative for psychiatric hospitalizationwithin the last 6 months or current use of antipsychotic medica-tion, (e) negative for current substance abuse or treatment, (f)normal wake–sleep cycle, and (g) not planning to initiate a newmedical treatment for their pain during the study period. Partici-pants were not asked to alter any current medical treatment, suchas medication or physical therapy. Eligibility also required theability to read, write, and speak English; the absence of signifi-
140 Broderick et al. Annals of Behavioral Medicine
Do Not
Cop
y
cant hearing or visual impairment; and the ability to come to theresearch site five times over the 24-day research protocol.
The Stony Brook University institutional review board ap-proved the study protocol, and each participant gave informedconsent. Participants were paid $150 for completion of the studyprotocol.
Materials
Paper diary technology. A DayRunner Organizer binder(17 cm × 11 cm × 3.5 cm) with case and belt strap heldpreprinted diary cards that were locked into the binder with plas-tic rings to prevent participants from removing the cards fromthe binder. The binder was unobtrusively instrumented to detectvia photo sensors (patent pending, invivodata, inc., Scotts Val-ley, CA) when the binder was opened and closed. A circuitboard recorded the time and date of each opening and closing ofthe binder to within 1-min resolution. These data were uploadedto a computer when participants returned to the laboratory. Al-though opening of the binder does not guarantee that a diary en-try was made, absence of an opening indicates that the entrycould not have been made at that time.
Extensive testing of this instrument was conducted to vali-date its ability to detect opening and closing under a range oflight and temperature conditions. Independent recordings ofopenings and closings were conducted in the field to verify ac-curacy of instrument time and date recording.
Diary assessment content. Comparable to typical researchpain diaries, pain, mood, setting, and current activities were re-corded at each assessment. A maximum of 20 items was com-pleted if the participant was experiencing pain at the time of theassessment. The items were taken from several instruments, in-cluding the Brief Pain Inventory (49), the Pain Discomfort Scale(50), and the Health Assessment Questionnaire (51).
The diary cards were labeled Morning Report for the 10:00a.m. assessment, Afternoon Report for the 4:00 p.m. assessment,and Evening Report for the 8:00 p.m. assessment. Each card wasnumbered 1 to 24 to indicate the day in the protocol. Thefixed-time assessment cards were placed in sequential order inthe binder for each week. A separate section at the back of thebinder, marked with a tab, contained additional diary cards with-out day and time labeling for use in reporting of pain exacerba-tions (which are not examined in this article).
Diary signaling device. To receive signals that it was timeto make a paper diary entry, participants were instructed to weara specially programmed digital wristwatch, originally designedas a medication reminder (Cadex Products, Ketchum, ID). Thiswas the only change in the paper diary implementation com-pared to Stone et al. (48). We programmed a schedule of signalsso that participants would be “beeped” by the watch at the onsetof the 30-min window surrounding the targeted reporting times(10:00 a.m., 4:00 p.m., 8:00 p.m.). Each beep lasted 60 sec, andthe message “scheduled diary entry” scrolled across the face ofthe watch while the watch beeped. The watch continued to sig-nal participants every 3 min indefinitely unless the watch was
manually shut off (by pressing a small button on the watch). Thewatch’s programming features were disabled so participantscould not alter these functions.
Questionnaires. We used the short form of the McGill PainQuestionnaire (52),awidelyused instrumentwithsatisfactoryre-liability and validity, to document participants’baseline sensory,affective, and evaluative components of their pain experience.
Design and Procedure
During their initial visit to the research laboratory, partici-pants provided informed consent; completed demographicitems and the McGill Pain Questionnaire; and, in a group set-ting, received instruction (approximately 45 min) about how andwhen (± 15 min of the targeted assessment time—correspond-ing to the 30-min window) to complete the pain diary cards. Atthis training, each participant received an instrumented diarywith cards for 7 days of fixed-time assessments plus 10 pain ex-acerbation reports. Participants were not informed of the instru-mentation in the binder. Participants were also given the prepro-grammed wristwatch and were instructed to wear it during allwaking hours. The protocol entailed completion of diary entriesfor 24 consecutive days at 10:00 a.m., 4:00 p.m., and 8:00 p.m.within a 30-min window. Participants were told that completionof the assessment at any point during these 30 min was accept-able. They were told to keep the diary closed except when mak-ing entries. In addition to the diary content mentioned earlier,participants were to record the day and time of each entry. Par-ticipants were also instructed to make a diary entry in a differentsection of the diary (i.e., not with the fixed-time diary cards)whenever they experienced a significant exacerbation of pain.Data from these participant-driven recordings are not addressedin this article. Participants were instructed to carry the diarythroughout the day.
Participants returned to the laboratory 2 to 3 days after theirinitial visit and training. These first few days of data collectionwere considered practice and were not included in the data anal-ysis. The “official” monitoring period began at the conclusion ofthis second visit and continued for 21 days or until Day 24,whichever came first. Participants returned to the laboratoryweekly during this period. During each visit, diaries were col-lected from participants and taken to another room. Used diarycards were removed and replaced with cards for the next 10 to12 days (to allow for data collection to continue, even if the par-ticipant did not get back to the laboratory within 7 days), and thebinder’s electronic data were uploaded to a computer. In addi-tion, a research assistant reviewed the past week’s data collec-tion with the participant (based only on the paper entries, not theinstrumented monitoring), problem-solved evident difficulties,and encouraged continued adherence to the protocol. The re-search assistant was not aware of the participant’s compliancewith the protocol from the instrumented data.
Debriefing. During the participants’ final visit to the labo-ratory (after the 3rd week of the diary protocol), they completed
Volume 26, Number 2, 2003 Diary Compliance 141
Do Not
Cop
y
a brief questionnaire pertaining to their experience with the di-ary and their estimated compliance with the sampling protocol.
Data preparation. Data records were generated for eachparticipant, listing each opening and closing of the diary binderand the self-reported date and time of each completed diary en-try. Three types of compliance with the fixed-time assessmentprotocol were generated: (a) verified compliance, (b) self-re-ported compliance, and (c) falsified compliance. Two versionsof each type of compliance were generated, one using the desig-nated 30-min window and the second using a more liberal90-min window. Verified compliance was defined as a record ofthe binder being opened (or closed) and a diary entry reportedlybeing made at any time within the designated 30- or 90-min win-dows surrounding the assessment time (10:00 a.m., 4:00 p.m.,8:00 p.m.). Self-reported compliance was defined as a time anddate written on the diary card that fell within the 30- and 90-minwindows. Falsified compliance was defined as entries that wereself-reported as completed within a protocol window, but thebinder record indicated that the binder was never open duringthe window. The denominator for all compliance estimates wasthe total number of fixed assessments scheduled for each partic-ipant minus the number of assessments that were excluded asdefined later in this article.
Data analysis. Compliance rates for each participant werecalculated for the entire 21-day monitoring period and sepa-rately for each 7-day period. For ease of interpretation, we re-port the average of the participants’ compliance rates. However,when testing for group differences in compliance (analysis ofvariance [ANOVA]) or across weeks for trends in compliance(repeated measures ANOVA), the arcsine transformation wasapplied to reduce the floor/ceiling effects and to reduce skew-ness in the distribution of rates. We used SAS version 8.0 fordata analysis.
Across 27 participants and 21 days, there were 1,613 as-sessment windows when the diary was to be completed. Therewere two types of binder openings that were ambiguous with re-gard to compliance. The first were instances when a scheduleddiary entry occurred during a time when the binder was left openfor more than 45 min. Because the binder was not opened andclosed for the entry, there was uncertainty about whether an en-try was made during this time. The second ambiguous type ofopenings were assessment windows that coincided with labora-tory visits. Because binders were opened and closed during alaboratory visit, it was not possible to determine if an openingwas due to the participant making a diary entry or to the researchstaff working with the binder. In both of these situations, theseassessment windows were excluded from the calculation ofcompliance—that is, removed from both the numerator and de-nominator.
RESULTS
Participants
Eighty-six adults responded to recruitment advertisements.Thirty of the screened respondents (35%) were ineligible ac-
cording to the criteria outlined earlier. Five (6%) of the respon-dents were not interested in volunteering for the study. Of the 51who were eligible to be scheduled, 18 (35%) were not scheduledbecause of scheduling conflicts or inability to be reached.Thirty-three adults were scheduled to begin the study. Five didnot arrive for the study (15%), and 1 participant (3%) did notcomplete the protocol. Some demographic information from thetelephone screening permitted a comparison of the 29 eligibleadults who did not participate with the 27 adults who did partici-pate. Age, gender, and average level of pain in the last weekwere compared. Only age differed, with nonparticipants beingyounger than the participants (42.7 years vs. 49.5 years), t(54) =2.57, p = .01.
Table 1 presents the demographic and pain characteristicsof the 27 participants who completed the protocol. The meanlevels of pain reported during screening were consistent withother chronic pain populations (1–3) and with themusculoskeletal pain sample used to validate the McGill PainQuestionnaire short form (4).
Compliance
Verified compliance. Using the data from the instrumenta-tion within the paper diary binder, we determined that the aver-age compliance for the 30-min window was 29% (see Table 2).The two highest compliance rates observed for individual par-ticipants were 72% and 62%, and 6 participants had compliancerates below 5%. When we allowed a 90-min window, the aver-age verified compliance rate was 39%.
Across all participants, there were 444 days (excluding lab-oratory visit days) on which diary entries should have beenmade. The binder was not opened at all on 99 (22%) of these
142 Broderick et al. Annals of Behavioral Medicine
TABLE 1Characteristics of the Sample
Age (M � SD) 50.2 � 8.9
% Female 44Education
% Graduated high school 63% Graduated college 37
% White 92Income (%)
Less than $20,000 19$20,000–$49,000 41$50,000–$75,000 19Greater than $75,000 22
Marital status (%)Never 11Married/ Living with partner 48Separated, divorced, widowed 41
days. Only 48% of participants opened the diary at least once onevery day of the study, and there was substantial variabilityacross participants, with 1 participant not opening the binder on82% of the days.
Self-reported compliance. Using the 30-min window (± 15min around each fixed assessment), participants’ self-reportedcompliance was 85%. Using the more liberal 90-min window,self-reported compliance was 91%.
Falsified compliance. The difference between self-re-ported compliance and instrument-verified compliance is therate of falsified compliance. For the 30-min window, 56% ofthe entries were falsified, and for the 90-min window, 52%were falsified.
As noted earlier, many participants did not open the binderat all on some days. Nevertheless, for the 99 days (out of the 444across participants) that the binder was not opened, on 96 (97%)of these days participants submitted at least 1 diary entry for thatday. Moreover, of the 288 possible entries on these 96 days, par-ticipants submitted 287 (99%) entries and reported that 272(95%) were completed within the 30-min window.
Factors affecting verified compliance. We computed acorrelation between initial level of pain (100 mm visual ana-logue scale) and verified compliance to determine if level ofpain could have a motivating or thwarting effect on compliance.No significant relation was observed (p > .30).
We compared verified compliance rates at the three sampletimes (10:00 a.m., 4:00 p.m., 8:00 p.m.) to determine if they sys-tematically varied. A repeated measures (time of day) ANOVAwas significant, F(2, 52) = 3.91, p < .03. Post hoc comparisonsindicated that the 10:00 a.m. and 4:00 p.m. compliance rates(32% and 31%, respectively) were each significantly greaterthan the 8:00 p.m. rate of 25%, t(52) = 2.55, p = .01, and t(52) =2.27, p = .03, respectively.
Verified compliance deteriorated significantly over the3-week reporting period for the 30-min window, F(2, 48) = 5.65,p < .007, and for the 90-min window, F(2, 48) = 3.85, p < .03.Table 3 displays the rates of compliance for each week. For bothcompliance windows, verified compliance dropped by almost10% from Week 1 to Week 2 (30-min window: 35% to 25%;90-min window: 44% to 35%) and then remained at the lowerlevel for the 3rd week.
End-of-study questionnaire. At the end of the protocol,during their last laboratory visit, participants answered a seriesof questions about their experience in the study. Ninety-threepercent rated the diary “very easy” or “extremely easy” to readand to complete. All participants rated their initial training inuse of the diary as “very clear” or “extremely clear.” Forty-sixpercent rated it “somewhat easy” to keep the diary with them at
Note. CI = confidence interval.aFifty-six were due to binders being left open ± 45 min overlapping with
an assessment window; 9 windows occurred during weekly laboratory vis-its. bFifty-nine were due to binders being left open more than 45 min over-lapping with an assessment window; 18 windows occurred during weeklylaboratory visits.
TABLE 3Rates of Compliance by Study Week
Time Frame Week 1: M (95% CI) Week 2: M (95% CI) Week 3: M (95% CI) p
Note. Means in the same row with different subscripts differ significantly: a,b, p < .01; c,d, p < .05. CI = confidence interval.aExclusions include targeted assessment times falling within period when binder was opened for more than 45 min or during laboratory visit.
Do Not
Cop
y
all times; 50% rated it “very easy” or ”extremely easy” to keepwith them.
Another question asked participants about how successfulthey were at making the daily entries at 10:00 a.m., 4:00 p.m.,and 8:00 p.m. Eighty-two percent rated themselves as “ex-tremely successful” or “very successful”; 11% gave a rating of“moderately successful,” and 7% gave a rating of “somewhatsuccessful.” To determine whether participants accurately esti-mated their compliance, we computed correlations of partici-pants’ ratings of how successful they were at adhering to the pro-tocol with verified and self-reported compliance for the 30-minwindow. As expected, the correlation between estimated suc-cess with the protocol and self-reported compliance was posi-tive and large (r = .57, p = .002). However, the correlation be-tween estimated success and verified compliance wassignificant in the wrong direction (r = –.43, p = .025), such thathigher estimates of success were actually associated with lowerverified compliance. Participants (n = 5) who estimated theircompliance as only somewhat or moderately successful had anaverage verified compliance of 54%, whereas those who esti-mated their success as very good or excellent (n = 22) had an av-erage verified compliance of 23%.
Comparison of signaled and unsignaled diary compliance.To determine whether signaling improves compliance, we com-
pared compliance rates from our earlier study (48), in which weused the same paper diary protocol but used no signaling, with thecompliance rates in this signaled diary protocol. It is important tonote that the patient selection criteria and recruitment methodswere identical inboth studies.Demographiccharacteristicsof thetwo samples were compared, and the only marginally significantdifference was sex, χ²(1, N = 65) = 2.77, p = .10. There were pro-portionately more men in the signaled diary sample than in theunsignaled sample, and therefore sex was treated as a covariate inthe following analyses. The compliance rates for the two samplesare presented in Table 4. The rates of self-reported compliance byparticipants did not differ between the two protocols; however,the diary protocol with signaling achieved significantly higherrates of verified compliance than the unsignaled protocol, F(1,62) = 13.15, p = .0006, for the 30-min window, and F(1, 62) =8.44, p = .005, for the 90-min window.
Comparison of unsignaled diary and electronic diary com-pliance. In our earlier study (48), we showed that an electronicdiary protocol with compliance-enhancing features (includingauditory signaling, small size, and weekly compliance feedbackbased on verified performance) yielded much better compliancerates than the protocol for the unsignaled diary group. Althoughwe observed a significant improvement for the signaled diarygroup in this study over the earlier unsignaled group, we alsowondered whether the signaled group compliance rates weresignificantly worse than the compliance-enhanced electronic di-ary group from our earlier study. If not, then from the perspec-tive of compliance there would be no need to suggest use of theelectronic diary. As with the prior set of analyses, demographiccharacteristics of the two groups were compared; the only sig-nificant group difference was for age, t(64) = 3.19, p < .01, andage was therefore included as a covariate. The verified compli-ance with the electronic diary (93.6%; 95% confidence interval:91.5–95.7) was significantly higher than with the signaled diary(29.1%; 95% confidence interval: 20.2–38.0), F(1, 63) =227.80, p < .0001.
DISCUSSION
This study was designed to evaluate, in a population of indi-viduals with chronic pain, compliance with a typical experiencesampling diary protocol using a signaling methodology. Previ-ous work in our laboratory with the same patient population, di-ary devices, and procedures, but absent the signaling feature, re-vealed extremely low levels of verified compliance (48). Thecurrent hypothesis was that signaling would assist patients in re-membering to make diary entries and result in improved compli-ance. Although significantly improved over the unsignaled pro-tocol, the observed compliance rates were in fact modest and notwhat could be considered satisfactory. Whereas the self-re-ported compliance of 80% to 90% is consistent with other com-pliance rates reported in the diary literature, verified compliancewas substantially less, ranging from 24% to 44% depending onthe week in the protocol and the size of the compliance window.These disappointing compliance rates were evident in this pro-tocol of three assessments per day even in the 1st week of the di-ary and therefore cannot be attributed to excessive sampling bur-den. The literature is replete with examples of protocols
144 Broderick et al. Annals of Behavioral Medicine
TABLE 4Rates of Compliance Across 21 Days for Signaled and Unsignaled Paper Diary Protocols
Time Frame Signaleda: M (95% CI) Unsignaledb: M (95% CI) pc
Note. CI = confidence interval.an = 27. bn = 40. cBased on an analysis of covariance predicting compliance (arcsine transformed) with sex treated as a covariate.
Do Not
Cop
y
requiring multiple daily assessments over weeks and evenmonths.
One might argue that the poor compliance observed in thisstudy was the result of poor motivation or other characteristicsof our participants. However, this explanation is unlikely giventhat these patients were from the same pool of patients (samestudy recruitment procedures, same time period, same paincharacteristics) as those who demonstrated 94% compliance inthe identical protocol using an electronic diary with compli-ance-enhancing features (48). Furthermore, the education andincome levels of our sample may also be somewhat higher thanthe average chronic pain patient, thus suggesting that we mightexpect higher rather than lower levels of compliance in thisstudy. A second explanation could be that, because our patientswere not receiving medical treatment in our protocol, perhapstheir motivation to provide accurate, compliant data was lessthan optimal because they expected no direct benefit from theirparticipation. This is plausible and should be explored in sub-sequent research. Finally, a third reason could be that our par-ticipants were inadequately trained in the diary protocol. Infact, our protocol training was probably more comprehensivethan many others, precisely because we wanted to maximizecompliance with the procedure. The patients attended a 45-mininitial training (standardized with overhead slides) in the use ofthe paper diary and watch and in the schedule of assessments.They came back into the laboratory 3 days later, and diary en-tries were reviewed, any impediments to timely recordingbased on their self-report were problem-solved, and encour-agement was given to adhere to the sampling protocol. Theyreturned to the laboratory each week over the subsequent21-day monitoring period, and similar review and encourage-ment procedures were implemented. Furthermore, the signal-ing procedure entailed reminder beeps that persisted every 3min after the first signal (which began 15 min prior to the pro-tocol assessment time) and could be terminated only by a par-ticipant action, although we cannot be sure about participants’compliance with wearing the watch. Thus, the compliance ob-tained in this study is probably better than that obtained in lessrigorously implemented diary studies (43,53–58). It appearsthat the task of in vivo data collection is a difficult one to ac-complish without the explicit accountability inherent in somemonitoring devices, such as the electronic diary protocol de-scribed earlier.
Another explanation that might be offered for the poorcompliance observed in this study is the method by which weverified compliance. Because a diary entry was coded as com-pliant only when there was evidence that the diary had beenopened during the assessment window, then if participants keptthe diary open most of the time, there would be no verified evi-dence of compliance. However, this can be ruled out as well be-cause our analysis strategy specifically excluded any assess-ment windows that fell during a period when the binder was leftopen for more than 45 min, and the number of such incidentswas small in relation to the total number of assessment win-dows. Thus, compliance was not affected negatively by partici-pants failing to keep the diary closed.
When we realized that signaling did not bring complianceto an acceptable level and that the range of compliance for indi-viduals was wide, we examined whether participants accuratelyestimated their compliance at the end of the study. If they could,then researchers and clinicians would have a way to identifynoncompliant respondents. We unexpectedly found that the par-ticipants who rated their compliance lowest in fact had the high-est levels of verified compliance, whereas the other participants,who rated their success with the protocol highly, had muchlower compliance. We can only speculate and draw on anecdotalobservations to explain this finding. The 1 participant whodropped out of this study did so because he felt his compliancewith the protocol was inadequate and he did not want to submitpoor-quality data. In fact, his compliance was the highest amongthe participants for the 1st week. Another participant in our pre-viously published unsignaled diary study dropped out for thesame stated reason. We speculate that individuals who are con-scientious and who are highly motivated to adhere strictly to theprotocol may be more objective and candid about their imper-fect compliance, whereas less motivated or conscientious indi-viduals evaluate their effort as sufficient even when it signifi-cantly deviates from the protocol. A less attractive explanationis that participants knowingly and purposefully misrepresenttheir behavior. In any case, it does not appear that participants’estimations of adherence are at all informative with respect toactual compliance.
In considering the emerging evidence of poor compliancein paper diaries, several factors should be considered. Not allsymptom reports are as time sensitive as others. For example, di-ary entries—even if they are completed up to 24 hr late—mayhave valid information when reporting highly discrete andmemorable events, such as an asthma attack or onset of a coldsore. Poor compliance with the timing of those diary entries maynot be as problematic as for other, less discrete events, such aschronic pain, mood, and medication use. To the extent that theevents to be reported occur frequently, vary over short time in-tervals, or may be forgotten or distorted, deviation from the timesampling protocol is likely to jeopardize the integrity of the data.This is particularly likely when the researcher is interested in ex-amining associations in close temporal proximity, as would bethe case in studies, for example, that are correlating ambulatoryblood pressure with self-reported concurrent location, activity,and stress levels. Our finding of the diary not being opened at allon 22% of the days, and yet almost all of the diary entries forthose days being completed, documents that patients are fre-quently making entries many hours or even days distant from thesampling moment.
This study and our previous work emphasize the impor-tance of objective verification of diary completion. In additionto date–time stamping methods offered by electronic diaries,there are other creative solutions to monitoring. End-of-day dia-ries can be mailed back by patients on a daily basis, and the post-mark can be used to verify timely completion of the diary(59–61). This solution limits verification to knowing that a diarywas not completed after the postmarked date. It does not ensurethat the diaries were completed as per the protocol, for example,
Volume 26, Number 2, 2003 Diary Compliance 145
Do Not
Cop
y
at the end of the day. For this reason, it is not a very useful strat-egy for monitoring protocols that require multiple reports in agiven day. Interactive voice response technology collects verbaland keypad-entered data via a telephone system linked to a com-puter. Patients and research participants can call the system fromany location, be walked through the assessment, and provide re-sponses by using the touchtone keys (62–65). The advantage ofthis system is that the diary does not have to be carried with theparticipant, multiple assessments can be administered in a day,and verification of the date and time of assessment is assured.However, participants must have access to a phone throughoutthe day. Finally, data collection via the Internet is beginning todevelop and has the same advantages and disadvantages as theinteractive voice response technology.
It is becoming increasingly tenuous to assume, without in-dependent verification, that time-sensitive data collection frompatients and research participants in their natural environment iscompliant with protocol. As shown in this study, actual compli-ance is unacceptably low and deteriorates after the 1st week ofdata collection. The evidence is compelling: Regardless ofwhether research uses reports of medication use, collection ofsaliva samples, or diary entries, patients significantly inflatetheir self-reports of compliance. This phenomenon transcendspatient characteristics, medical conditions, types of data collec-tion, and sampling burden. We continue to be optimistic thatmethods can be developed to achieve acceptable and sustainablecompliance by our research participants and patients. The evi-dence certainly suggests that patients want to appear compliantand that good compliance that does not deteriorate over thelength of the protocol can be achieved with some electronic datacollection methods (66–67). However, paper diary methodolo-gies that rely on self-reported compliance appear seriouslyflawed. This seems to be the inevitable conclusion, but an unfor-tunate one, for researchers and clinicians who rely on this rela-tively inexpensive approach to data collection.
REFERENCES(1) Verbrugge L: Health diaries. Medical Care. 1980, 18:73–95.(2) Eisenhower D, Mathiowetz N, Morganstein D: Recall error:
Sources and bias reduction techniques. In Biemer P, Groves L,Lyberg N, Mathiowetz N, Sudman S (eds), Measurement Er-rors in Surveys. New York: Wiley, 1991, 127–144.
(3) Bradburn N, Rips L, Shevell S: Answering autobiographicalquestions: The impact of memory and inference on surveys.Science. 1987, April 10:157–161.
(4) Ross M: Relation of implicit theories to the construction of per-sonal histories. Psychological Review. 1989, 96:341–357.
(5) Gorin A, Stone A: Recall biases and cognitive errors in retro-spective self-reports: A call for momentary assessments. InBaum A, Revenson T, Singer J (eds), Handbook of Health Psy-chology. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.,2001, 405–413.
(6) Schwarz N, Sudman S: Autobiographical memory and the va-lidity of retrospective reports. New York: Springer-Verlag,1994.
(7) Goodwin A, Sher K: Effects of induced mood on diagnostic in-terviewing: Evidence for a mood and memory effect. Psycho-logical Assessment. 1993, 5:197–202.
(8) Clark DM, Teasdale JD: Diurnal variation in clinical depressionand accessibility of memories of positive and negative experi-ences. Journal of Abnormal Psychology. 1982, 91:87–95
(9) Harkins S, Price D, Braith J: Effects of extraversion andneuroticism on experimental pain, clinical pain, and illness be-havior. Pain. 1989, 36:209–218.
(10) Wade J, Dougherty L, Hart R, Cook D: Patterns of normal per-sonality structure among chronic pain patients. Pain. 1992,48:37–43.
(11) Eich E, Reeves JL, Jaeger B, Graff-Radford SB: Memory forpain: Relation between past and present pain intensity. Pain.1985, 23:375–380.
(12) Smith W, Safer M: Effects of present pain level on recall ofchronic pain and medication use. Pain. 1993, 55:355–361.
(13) Bradburn N: Temporal representation and event dating. InStone A, Turkkan J, Bachrach C, Jobe J, Kurtzman H, Cain V(eds), The Science of Self-Report: Implications for Researchand Practice. Mahwah, NJ: Lawrence Erlbaum Associates,Inc., 2000, 49–61.
(14) Shiffman S: Real-time self-report of momentary states in thenatural environment: Computerized ecological momentary as-sessment. In Stone A, Turkkan J, Bachrach C, Jobe J, KurtzmanH, Cain V (eds), The Science of Self-Report: Implications forResearch and Practice. Mahwah, NJ: Lawrence Erlbaum As-sociates, Inc., 2000, 277–296.
(15) deVries M, Dijkman-Caes C, Deslespaul P: The sampling ofexperience: A method of co-measuring the co-occurrence ofanxiety and depression in daily life. In Maser J, Cloninger C(eds), Comorbidity of Mood and Anxiety Disorders. Washing-ton, DC: American Psychiatric Press, 1990, 869.
(16) Delespaul P: Assessing Schizophrenia in Daily Life: The Expe-rience Sampling Method. Maastricht, The Netherlands:Universitaire Pers Maastricht, 1995.
(17) Stone A, Shiffman S: Ecological momentary assessment(EMA) in behavioral medicine. Annals of Behavioral Medicine.1994, 16:199–202.
(18) Shiffman S, Hufford M, Hickcox M, et al.: Remember that? Acomparison of realtime vs. retrospective recall of smokinglapses. Journal of Consulting and Clinical Psychology. 1997,65:292–300.
(19) Bornstein RF: Implicit and self-attributed dependencystrivings: Differential relationships to laboratory and field mea-sures of help seeking. Journal of Personality and Social Psy-chology. 1998, 75:778–787.
(20) Drossman DA, Leserman J, Li Z, et al.: Effects of coping onhealth outcome among women with gastrointestinal disorders.Psychosomatic Medicine. 2000, 62:309–317.
(21) Feldman SI, Downey G, Schaffer-Neitz R: Pain, negativemood, and perceived support in chronic pain patients: A dailydiary study of people with reflex sympathetic dystrophy syn-drome. Journal of Consulting and Clinical Psychology. 1999,67:776–785.
(22) Diener E, Fujita F: Resources, personal strivings, and subjec-tive well-being: A nomothetic and idiographic approach. Jour-nal of Personality and Social Psychology. 1995, 68:926–935.
(23) de Wit R, van Dam F, Hanneman M, et al.: Evaluation of the useof a pain diary in chronic cancer pain patients at home. Pain.1999, 79:89–99.
(24) Richter JE, Kahrilas PJ, Sontag SJ, et al.: Comparinglansoprazole and omeprazole in onset of heartburn relief: Resultsof a randomized, controlled trial in erosive esophagitis patients.American Journal of Gastroenterology. 2001, 96:3089–3098.
146 Broderick et al. Annals of Behavioral Medicine
Do Not
Cop
y
(25) Affleck G, Tennen H, Urrows S, Higgins P: Individual differ-ences in the day-to-day experience of chronic pain: A prospec-tive daily study of rheumatoid arthritis patients. Health Psy-chology. 1991, 10:419–426.
(26) Farnworth L: Time use and leisure occupations of young of-fenders. American Journal of Occupational Therapy. 2000,54:315–325.
(27) Carstensen LL, Pasupathi M, Mayr U, Nesselroade JR: Emo-tional experience in everyday life across the adult life span. Jour-nal of Personality and Social Psychology. 2000, 79:644–655.
(28) Hillbrand M, Waite BM, Miller DS, Spitz RT, Lingswiler VM:Serum cholesterol concentrations and mood states in violentpsychiatric patients: An experience sampling study. Journal ofBehavioral Medicine. 2000, 23:519–529.
(29) Shiu AT: The part-time student role: Implications for the emo-tional experience of managing multiple roles amongst HongKong public health nurses. Nurse Education Today. 1999,19:188–198.
(30) van Eck M, Berkhof H, Nicolson N, Sulon J: The effects of per-ceived stress, traits, mood states, and stressful daily events onsalivary cortisol. Psychosomatic Medicine. 1996, 58:447–458.
(31) Frable DE, Platt L, Hoey S: Concealable stigmas and positiveself-perceptions: Feeling better around similar others. Journalof Personality and Social Psychology. 1998, 74:909–922.
(32) Gauvin L, Rejeski WJ, Norris JL: A naturalistic study of the im-pact of acute physical activity on feeling states and affect inwomen. Health Psychology. 1996, 15:391–397.
(33) Cruise CE, Broderick J, Porter L, Kaell A, Stone AA: Reactiveeffects of diary self-assessment in chronic pain patients. Pain.1996, 67:253–258.
(34) Ockenfels MC, Porter L, Smyth J, et al.: Effect of chronic stressassociated with unemployment on salivary cortisol: Overallcortisol levels, diurnal rhythm, and acute stress reactivity. Psy-chosomatic Medicine. 1995, 57:460–467.
(35) Richards MH, Duckett E: The relationship of maternal employ-ment to early adolescent daily experience with and without par-ents. Child Development. 1994, 65:225–236.
(36) Turner S, Smedley J, Cherry N: Estimating occupational healthevents in workers with asthma or diabetes: A comparison of di-ary and snapshot methods. Occupational Medicine. 2001,51:325–331.
(37) Merlo J, Berglund G, Wirfalt E, et al.: Self-administered ques-tionnaire compared with a personal diary for assessment of cur-rent use of hormone therapy: An analysis of 16,060 women.American Journal of Epidemiology. 2000, 152:788–792.
(38) Maunsell E, Allard P, Dorval M, Labbe J: A brief pain diary forambulatory patients with advanced cancer. Cancer. 2000,88:2387–2397.
(39) Urquhart J: Patient non-compliance with drug regimens: Mea-surement, clinical correlates, economic impact. EuropeanHeart Journal. 1996, 17(Suppl. A):8–15.
(40) Urquhart J, De Klerk E: Contending paradigms for the interpre-tation of data on patient compliance with therapeutic drug regi-mens. Statistics in Medicine. 1998, 17:251–267, 387–389.
(41) Mallion JM, Baguet JP, Siche JP, Tremel F, de Gaudemaris R:Compliance, electronic monitoring and antihypertensive drugs.Journal of Hypertension. 1998, 16(Suppl. 1):S75–S79.
(42) Boudes P: Drug compliance in therapeutic trials: A review.Controlled Clinical Trials. 1998, 19:257–268.
(43) Rand CS, Wise RA, Nides M, et al.: Metered-dose inhaler ad-herence in a clinical trial. American Review of Respiratory Dis-ease. 1992, 146:1559–1564.
(44) Spector SL, Kinsman R, Mawhinney H, et al.: Compliance ofpatients with asthma with an experimental aerosolized medica-tion: Implications for controlled clinical trials. Journal of Al-lergy and Clinical Immunology. 1986, 77(1, Pt. 1):65–70.
(45) Mazze RS, Shamoon H, Pasmantier R, et al.: Reliability ofblood glucose monitoring by patients with diabetes mellitus.American Journal of Medicine. 1984, 77:211–217.
(46) Nordmann A, Frach B, Walker T, Martina B, Battegay E: Com-parison of self-reported home blood pressure measurementswith automatically stored values and ambulatory blood pres-sure. Blood Pressure. 2000, 9:200–205.
(47) Kudielka B, Broderick J, Kirschbaum C: Compliance with sa-liva sampling protocols: Electronic monitoring reveals invalidcortisol daytime profiles in noncompliant subjects. Psychoso-matic Medicine (in press).
(48) Stone A, Shiffman S, Schwartz J, Broderick J, Hufford M: Pa-tient compliance with paper diaries. British Medical Journal.2002, 324:1193–1194.
(49) McDowell I, Newell C: Measuring health: A guide to ratingscales and questionnaires (2nd Ed.). New York: Oxford Uni-versity Press, 1996.
(50) Jensen M, Karoly P, Harris P: Assessing the affective compo-nent of chronic pain: Development of the Pain DiscomfortScale. Journal of Psychosomatic Research. 1991,35:149–154.
(51) Ramey D, Raynauld J-P, Fries J: The Health Assessment Ques-tionnaire 1992: Status and review. Arthritis Care and Research.1992, 5:119–129.
(52) Melzack R: The short-form McGill Pain Questionnaire. Pain.1987, 30:191–197.
(53) Mawhinney H, Spector SL, Kinsman RA, et al.: Compliance inclinical trials of two nonbronchodilator, antiasthma medica-tions. Annals of Allergy. 1991, 66:294–299.
(54) Coutts JA, Gibson NA, Paton JY: Measuring compliance withinhaled medication in asthma. Archives of Disease in Child-hood. 1992, 67:332–333.
(55) Vanhove GF, Schapiro JM, Winters MA, Merigan TC, BlaschkeTF: Patient compliance and drug failure in protease inhibitormonotherapy. Journal of the American Medical Association.1996, 276:1955–1956.
(56) Paterson DL, Swindells S, Mohr J, et al.: Adherence to proteaseinhibitor therapy and outcomes in patients with HIV infection.Annals of Internal Medicine. 2000, 133:21–30.
(57) Burney KD, Krishnan K, Ruffin MT, Zhang D, Brenner DE:Adherence to single daily dose of aspirin in achemoprevention trial. An evaluation of self-report and micro-electronic monitoring. Archives of Family Medicine. 1996,5:297–300.
(58) Pullar T, Kumar S, Feely M: Compliance in clinical trials. An-nals of the Rheumatic Diseases. 1989, 48:871–875.
(59) Affleck G, Tennen H, Keefe FJ, et al.: Everyday life withosteoarthritis or rheumatoid arthritis: Independent effects ofdisease and gender on daily pain, mood, and coping. Pain.1999, 83:601–609.
(60) Schanberg LE, Sandstrom MJ, Starr K, et al.: The relationshipof daily mood and stressful events to symptoms in juvenilerheumatic disease. Arthritis Care and Research. 2000,13:33–41.
(61) Carney MA, Armeli S, Tennen H, Affleck G, O’Neil TP: Posi-tive and negative daily events, perceived stress, and alcohol use:A diary study. Journal of Consulting and Clinical Psychology.2000, 68:788–798.
Volume 26, Number 2, 2003 Diary Compliance 147
Do Not
Cop
y
(62) Piette J: Interactive voice response systems in the diagnosis andmanagement of chronic disease. American Journal of ManagedCare. 2000, 6:817–827.
(63) Bardone AM, Krahn DD, Goodman BM, Searles JS: Using in-teractive voice response technology and timeline follow-backmethodology in studying binge eating and drinking behavior:Different answers to different forms of the same question? Ad-dictive Behaviors. 2000, 25:1–11.
(64) Searles JS, Perrine MW, Mundt JC, Helzer JE: Self-report ofdrinking using touch-tone telephone: Extending the limits of re-liable daily contact. Journal of Studies on Alcohol. 1995,56:375–382.
(65) Perrine MW, Mundt JC, Searles JS, Lester LS: Validation ofdaily self-reported alcohol consumption using interactive voiceresponse (IVR) technology. Journal of Studies on Alcohol.1995, 56:487–490.
(66) Stone AA, Shiffman S, Schwartz JE, Broderick JE, HuffordMR: Patient compliance with paper and electronic diaries. Con-trolled Clinical Trials (in press).
(67) Jamison RN, Raymond SA, Levine JG, et al.: Electronic diariesfor monitoring chronic pain: 1-year validation study. Pain.2001, 91:277–285.
148 Broderick et al. Annals of Behavioral Medicine
