Data Cleaning and Verification


These steps should be performed immediately after the subject leaves, if at all possible. They should definitely be performed the same day as the data collection session. Performing these steps is required to know if the data collection has been successful.

Please note: No files should be manually edited except the notes file.


1. Copy all data to the appropriate directory.

This should have already been done after the data collection (see instructions). You should have a directory on the desktop of the [which?] laptop with this format:
    • SessionX
      • [SessionX]-notes.txt
      • SubjectData.xml
      • wockets
      • mites
      • othersensors
      • merged
      • annotation

Here is a detailed description of all files files and naming conventions.



2. Use the audio recordings to create an activity annotation file.
  • a. When the program starts, the following window should pop up:
annotator_start_screen.JPG

  • b. If you are using a protocol different from the one provided with the software, make sure to copy and paste the ActivityLabelsRealtime.xml file into wockets folder.
annotator_1.JPG

  • c. Click Browse after Session Path. Provide a valid path where your session files are located. Click Ok. For example:
    • My Computer>EXT_HD(D:)>Data>Data for Merging>Sep2809>
annotator_2.JPG
      • Remarks:
        • Don't forget before starting the application to copy the audio files (.wav and .msv) to the appropriate directory as indicated in the data aggregation section: (...\session<date>\annotation\voice).
        • The program expects to see both types of audio files in the same location. Besides, it also expects that the number of files .wav and .msv match. If the files number doesn’t match, the program will continue running but it will send a message warning about the mistmatch.
        • If the program doesn't find the audio files in the appropriate location, it will not launch the annotation interface.


  • d. Click Start.
  • annotator_3.JPG


  • e. If the files are loaded successfully, you will see the following screen:
audioAnnotator_screen_1.jpg
.
  • f. Select the postures and ambulations.


  • h. When you are done, click on the Generate Xml button. This button will generate the "AnnotationIntervals.xml" file in the "...\session<date>\annotation\audioannotation" directory. Check that this file is generated. The merger and viewer software will read this file to load the time stamped audio annotations.




3. Merge all the data.

You now run software that will merge all the time synchronized data sources into a single file and create additional files used the graphing program.

Step 1: Click Wockets Data Merger on Mini computer desktop
Step 2: A window will popup that allows you to select a directory. Click Browse.
merger_1.JPG


Step 3: Click My Computer> EXT_HD(D:)> Data > Data for Merging> Aug1209> Ok
merger_2.JPG
Step 4: Start the merge process. Click convert.
merger_3.JPG

Step 5: Conversion log appears.
merger_4.JPG

Step 6: Once the merge process is completed. You will be prompted to Done. Click Done.
Step 7: Make a note of any warning messages you received in the notes file in the main data directory.
Step 8: Check that the software created merged csv files for all your sensors in the merged directory.
merger_5.jpg




4. Graph the summary data using specialized graphing program.

The only way to really know that the data are ok is to graph the various data and ensure that (1) it exists and (2) it is synchronized.
  • a. Open the Wockets Data Viewer.
  • b. Choose from the File Menu, open. Open the folder that you created.
  • dataviewer.jpg
  • c. This will load the data in the visualizer.
  • d. Wait patiently, it may take [how long] to load the data at first. There is a lot of information!


Perform a visual inspection

  • Missing data
    For each sensor type, check that there is no obvious and unexpected missing data for the duration of the experiment. If there does appear to be missing data, add approximate start/stop time and additional possible explanations to the notes file in the data directory ([name]-notes.txt).
    • Select all the MITes. Scan and look for major gaps or sampling rate problems (see sampling rate line). Also check that the labels of activities are correct.
    • Next, select the Wockets and Zephyr accelerometer, Actigraph(s) and phone accelerometer. Scan and look for major gaps. Also, scan through checking that annotation labels have no major errors.
    • Next, select MITes HR, Zephyr HR, Oxycon HR, Oxycon VO2. Scan and look for major gaps or problems.
    • Next, select Sensewear and the other devices.

  • Synchronization (quick check)
    Next check the synchronization between the sensors. Look for any major time offsets between the sensors, due to a clock being set incorrectly or a daylight savings problem. Also, check that the labels and the data appear to be in sync and make sense. You can use the manual sycnhronization markers in the datasets (the quick jumps seperated by stillness you had the subject perform at the beginning and the end) to see that there are no major problems.
    • Select Actigraph(s). Use the sync activities at both beginning and end of data collection session to determine actigraphs offset, if any. Record in manual annotation doc file. Check that the activity labels and the actigraph values are in sync.
    • Select an Actigraph, hip MITes, phone accelerometer(s), Zephyr accelermeter. Use the sync activities at both beginning and end of data collection session to determine the offsets of each sensor, as best as possible, relative to the hip MITes. Record in manual annotation doc file.
    • Select the MITes HR, Zephyr HR, Oxycon HR, and Oxycon VO2. Look for any major problems with syncing the HRs and VO2 (due to change in daylight savings time or a problem with the Oxycon sync button procedure). Record in manual annotation file.
    • Select Sensewear and the other devices and check for any syncing problem.



5. Graph the summary data using Excel.

The merge process creates a summary csv file with 1s epoch data. This is a key file, so we want to be sure it looks ok.

  • a. In the data directory, open the SummaryData.csv file. Double clicking it should open it in Excel.

  • b. Plot the Actigraph(s) data. Use the sync activities at both beginning and end of data collection session to determine actigraphs offset. If there is any offset, record it in [name]-notes.txt file.

  • c. Plot the Actigraph, hip MITes, phone accelerometer(s), Zephyr accelermeter. Use the sync activities at both beginning and end of data collection session to determine the offsets of each sensor, as best as possible, relative to the hip MITes. If there is any offset, record it in [name]-notes.txt file.
  • d. Plot the MITes HR, Zephyr HR, Oxycon HR, and Oxycon VO2. Look for any major problems with syncing the HRs and VO2 (due to change in daylight savings time or a problem with the Oxycon sync button procedure). If there is any problem, record it in [name]-notes.txt file.

  • e. Plot Sensewear and the other devices and check for any syncing problem. [Add detail here]. If there is any syncing problem, record it in [name]-notes.txt file.


6. Verify the quality of the data.

Use the "Quality Assessment Check List" to verify the quality of the data. All observations must be writing in the notes file. This step should be perform no later than 2 days after the data was collected. The latest quality assessment check list can be downloaded in the following link:






7. Backup the data.

Burn the data directory to a DVD. Label the DVD and put it somewhere safe.



8. Send the data to MIT for the 2nd pass verification.

  • a. Zip the data directory, rename to [name].zip-rename, and email to the MIT team. Indicate in the email that:
    • Visually data looks good. Note anything unusual that happened
    • Visually data looks in sync. Note anything unusual you saw.

  • b. The MIT team will check the data. If you have not heard back within 3 days from MIT that the data look good, write again and ask about the status of the check. Consider the data collection session not complete until you have received this final confirmation!