App Events in Private

Each instance of the Private shell provides a variable called Events that contains the data that is available in this context. Events is a list of event instances of different types. One of the main kinds of events is the App event. These events are generated when a participant is using the Unforgettable Research Services’ app.

The app collects accelerometry, obfuscated audio, GPS and battery level by default. A user can also choose to collect images and raw audio files (and may turn off the acelerometry, obfuscated audio and GPS if they desire).

In addition, however, the Unforgettable system augments the raw data using several custom algorithms and online services. For instance, machine learning classifiers are run on the obfuscated audio segments to tag events with audio_voice, audio_traffic etc.

At the bottom of the post is an example of a full App event. There is a lot of information, some of which is unlikely to be of relevance to researchers. In what follows, I will run through some of the most used fields.

Each event in the Unforgettable system has a type that you can access using e.type where e is an event. For App events this type is “__App__”. To extract all of the App events available in the current context, you could use the following code:

AppEvents = [e for e in Events if e.type == "__App__"]

Like all events, App events have time stamps that record the when the event started and ended:

'StartDateTime': '2019-06-16T14:00:00Z',
'StartDateTimeLocal': '2019-06-17T00:00:00Z',
'EndDateTime': '2019-06-16T14:59:59Z',
'EndDateTimeLocal': '2019-06-17T00:59:59Z',

App events are always one hour in duration (other events may have different durations). StartDateTime and EndDateTime capture the Coordinated Universal Time (known as UTC time), while StartDateTimeLocal and EndDateTimeLocal capture the time in the time zone in which the event took place.

All events also have a randomly generated code to identify the user:

'UserId': '2f50d2d3ad5a5193945b47889a5c9a88f7a4ff70f1297973107911432a067b17',

and another one to identify the event:

'id': 'ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6::ffc4284d-7aa3-415e-a3dd-b9125c556388',

This code becomes important if one refers to one event in another event. For instance, in some memory experiments we have conducted, we ask participants about events from their lives. In this case, there will be an event that stores the experimental data. One of the fields in this event is the App event which was being queried in a given trial. To get all of the information associated with that event one can search for the event id.

Most App events also have a latitude and longitude:

'latitude': -37.792988,
'longitude': 144.988911,

Although most App events tend to have GPS coordinates, there are times when they are not recorded – either because a user has not given the App permission or because the GPS is temporarily disconnected – for instance when a user puts their phone in flight mode. When analysing latitude and longitude, therefore it is important to check that they exist. For instance, to extract the latitudes from Events you might use code like this:

Latitudes = [e.latitude for e in Events if e.hasField ("latitude")]

Each App event provides a single latitude and a single longitude for each hour. Typically, though the app will collect multiple latitudes and longitudes. By default the app records a position every 10 minutes. To calculate the latitude and longitude for the hour the system takes the median latitude and longitude and then finds the GPS location from the set that is closest to this position. In this way, the summary position for the hour is always a location to which the user had been. If one were just to take the median latitude and longitude as the summary point it is possibly for the summary to be a place where the user was never at. If, however, you would like to work with the individual GPS points these are available in the GpsLocations field:

 'GpsLocations': [{'lat': -37.7929697,
                      'lon': 144.9888915,
                      'time': '2019-06-16 14:01:25Z',
                      'time_local': '2019-06-17 00:01:25'},
                     {'lat': -37.7929776,
                      'lon': 144.988879,
                      'time': '2019-06-16 14:12:15Z',
                      'time_local': '2019-06-17 00:12:15'},
                     {'lat': -37.792762010358274,
                      'lon': 144.98914819210768,
                      'time': '2019-06-16 14:22:17Z',
                      'time_local': '2019-06-17 00:22:17'},
                     {'lat': -37.7930073,
                      'lon': 144.9889067,
                      'time': '2019-06-16 14:32:12Z',
                      'time_local': '2019-06-17 00:32:12'},
                     {'lat': -37.7929881,
                      'lon': 144.9889107,
                      'time': '2019-06-16 14:42:15Z',
                      'time_local': '2019-06-17 00:42:15'},
                     {'lat': -37.7929881,
                      'lon': 144.9889107,
                      'time': '2019-06-16 14:52:13Z',
                      'time_local': '2019-06-17 00:52:13'}],

and the total distance travelled during the hour can be found using the Kilometers field:

'Kilometers': 0.072,

Unforgettable also queries location databases to turn the location into an address that appears in the Name field as a list of strings:

'Name': ['["Clifton Hill Primary School, 185, Gold Street, Clifton Hill, City '
             'of Yarra, Victoria, 3068, Australia", "Clifton Hill Primary '
             'School", "185", "Australia", "City of Yarra", "Clifton Hill", '
             '"Victoria", "3068", "Gold Street"]'],

The Unforgettable website provides users with a search interface to their data. This allows them to use the system as a memory prosthetic and to quickly find any data they my wish to delete from the system. To facilitate retrieval several keywords are added to each event. These keywords are also often useful for analysis.:

'Keywords': ['Monday',
                 'June',
                 '2019',
                 'Night',
                 'Cloudy',
                 'autumn',
                 'full',
                 'audio_home',
                 'audio_street'],

Typically, there will be tags indicating the day-of-week (e.g. Monday), the month (e.g. June) and the year (e.g. 2019). In addition, there are tags for the general time of day (Morning, 6am – 12 noon, Afternoon, 12 noon – 6pm, Evening, 6pm-9 pm or Night, 9pm – 6am) and the season (autumn, summer, spring or winter).

Keywords also contains a description of the weather that occurred during that hour. This description is a shortened version of the description that appears in the Weather field (see the description of the Weather field below). The possible values are Cloudy, Rain, Overcast, Windy, Drizzle and Clear.

There are also tags for the phase of the moon. The possible values are waning_crescent, last_quarter, waxing_gibbous, new_moon, full, waxing_crescent, first_quarter and waning_gibbous.

Finally, Keywords also contains a set of tags that are generated by machine classifiers that are run on the obfuscated audio. These include audio_voice, audio_street, audio_car, audio_cafe and audio_home.

The Weather field contains a brief description of the weather for that hour and this description appears in the Keywords as well. The Weather can be:

Clear
Dangerously Windy
Dangerously Windy and Mostly Cloudy
Dangerously Windy and Partly Cloudy
Drizzle
Heavy Rain
Humid
Humid and Mostly Cloudy
Humid and Partly Cloudy
Light Rain
Light Rain and Windy
Mostly Cloudy
Overcast
Partly Cloudy
Possible Drizzle
Possible Drizzle and Dangerously Windy
Possible Drizzle and Windy
Possible Light Rain
Possible Light Rain and Dangerously Windy
Possible Light Rain and Windy
Rain
Rain and Windy
Windy
Windy and Mostly Cloudy
Windy and Overcast
Windy and Partly Cloudy

One can access the temperature during that hour with the Temperature field:

'Temperature': 10.05,

There are three kinds of image data that the Unforgettable system can collect. If the user enables the option, the app will take pictures at regular intervals. However, managing the privacy of these images is difficult as it common for them to capture people unintentionally. Consequently, this is no longer enabled by default. Another alternative is to use Streetview images from Google. Unforgettable provides two versions – a thumbnail and a full image:

'StreetViewImage': 'https://s3-us-west-1.amazonaws.com/unforgettable-dev-usw1/userhome/ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6/2019/06/08/streetview_full_4a28b74a-1608-448b-b8e4-7831b6d5f858.jpeg',
    'StreetViewThumbnail': 'https://s3-us-west-1.amazonaws.com/unforgettable-dev-usw1/userhome/ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6/2019/06/08/streetview_thumb_4a28b74a-1608-448b-b8e4-7831b6d5f858.jpeg',

Finally, the app will record images that users upload in the UserImages field:

'UserImages': [],

Battery life can be a significant issue for participants depending on the type of phone they have and the their usage patterns. If data is missing then it might be because a user’s phone was running out of charge or because they were disabling the app to reduce battery usage. You can can an indication of whether this might have happened by analysing the battery level:

'BatteryLevel': 66,

One never knows when it might be important to know what the phase of the moon was during an event 😉 and so we provide:

'MoonAge': 14.0,
'MoonIllumination': 0.99,

App events also prove fields to access raw data that is retained. The streams that are retained are Accelerometry, AudioProcessed, Battery, Gps. The number of data points retain is stores in the AccelerometryCount, AudioProcessedCount, BatteryCount and LocationCount fields. The names of the files that store the raw data appear in AccelerometryDataFiles, AudioProcessedDataFiles, BatteryDataFiles and GpsDataFiles. And there are also file iterators to make processing these files more convenient – AccelerometryDataFilesItr, AudioProcessedDataFilesItr, BatteryDataFilesItr and GpsDataFilesItr. I’ll do a separate post to discuss what is in these files.

So that covers most of the information available in App events. Below is a sample of an entire event:

   {'AccelerometryCount': 5,
    'AccelerometryDataFiles': [{'filepath': '/data/ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6/2019/06/16/accel_20190616141000Z_b3944396-b87e-4b29-9acc-a05d00ed868f.bin',
                                'type': 'localfs'},
                               {'filepath': '/data/ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6/2019/06/16/accel_20190616142000Z_0983b4e0-8cab-4d29-bd78-e33278c0095f.bin',
                                'type': 'localfs'},
                               {'filepath': '/data/ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6/2019/06/16/accel_20190616143001Z_1503e036-bd86-4a29-87e3-b1ff73ec7929.bin',
                                'type': 'localfs'},
                               {'filepath': '/data/ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6/2019/06/16/accel_20190616144001Z_0e8df858-c8b3-4af7-ba45-020af013dfaa.bin',
                                'type': 'localfs'},
                               {'filepath': '/data/ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6/2019/06/16/accel_20190616145001Z_c2377e54-bc6d-40a3-b634-a325ce0a6c27.bin',
                                'type': 'localfs'}],
    'AccelerometryDataFilesItr': <Private.file_iterator.FileIterator object at 0x7f4d4ab4a898>,
    'AudioProcessedCount': 5,
    'AudioProcessedDataFiles': [{'filepath': '/data/ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6/2019/06/16/audio_20190616141007Z_28868a90-f7bd-4ad5-9264-e35c9b36e109.mfcc',
                                 'type': 'localfs'},
                                {'filepath': '/data/ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6/2019/06/16/audio_20190616142007Z_53f62074-5bc5-4902-9f5e-841d25e3684f.mfcc',
                                 'type': 'localfs'},
                                {'filepath': '/data/ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6/2019/06/16/audio_20190616143008Z_020d149b-8e99-485f-9337-7b43d9096169.mfcc',
                                 'type': 'localfs'},
                                {'filepath': '/data/ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6/2019/06/16/audio_20190616144008Z_6c9a97cf-8d2f-454b-8b21-be03f6949968.mfcc',
                                 'type': 'localfs'},
                                {'filepath': '/data/ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6/2019/06/16/audio_20190616145008Z_87feceae-29ca-4b1e-82f2-99621a652ebb.mfcc',
                                 'type': 'localfs'}],
    'AudioProcessedDataFilesItr': <Private.file_iterator.FileIterator object at 0x7f4d4ab4ab38>,
    'BatteryCount': 5,
    'BatteryDataFiles': [{'filepath': '/data/ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6/2019/06/16/battery_20190616140000Z_ffc4284d-7aa3-415e-a3dd-b9125c556388.csv',
                          'type': 'localfs'}],
    'BatteryDataFilesItr': <Private.file_iterator.FileIterator object at 0x7f4d4ab4ad68>,
    'BatteryLevel': 66,
    'EndDateTime': '2019-06-16T14:59:59Z',
    'EndDateTimeLocal': '2019-06-17T00:59:59Z',
    'GpsDataFiles': [{'filepath': '/data/ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6/2019/06/16/location_ffc4284d-7aa3-415e-a3dd-b9125c556388.csv',
                      'type': 'localfs'}],
    'GpsDataFilesItr': <Private.file_iterator.FileIterator object at 0x7f4d4ab4a5f8>,
    'GpsLocations': [{'lat': -37.7929697,
                      'lon': 144.9888915,
                      'time': '2019-06-16 14:01:25Z',
                      'time_local': '2019-06-17 00:01:25'},
                     {'lat': -37.7929776,
                      'lon': 144.988879,
                      'time': '2019-06-16 14:12:15Z',
                      'time_local': '2019-06-17 00:12:15'},
                     {'lat': -37.792762010358274,
                      'lon': 144.98914819210768,
                      'time': '2019-06-16 14:22:17Z',
                      'time_local': '2019-06-17 00:22:17'},
                     {'lat': -37.7930073,
                      'lon': 144.9889067,
                      'time': '2019-06-16 14:32:12Z',
                      'time_local': '2019-06-17 00:32:12'},
                     {'lat': -37.7929881,
                      'lon': 144.9889107,
                      'time': '2019-06-16 14:42:15Z',
                      'time_local': '2019-06-17 00:42:15'},
                     {'lat': -37.7929881,
                      'lon': 144.9889107,
                      'time': '2019-06-16 14:52:13Z',
                      'time_local': '2019-06-17 00:52:13'}],
    'Keywords': ['Monday',
                 'June',
                 '2019',
                 'Night',
                 'Cloudy',
                 'autumn',
                 'full',
                 'audio_home',
                 'audio_street'],
    'Kilometers': 0.072,
    'LocationCount': 6,
    'MoonAge': 14.0,
    'MoonIllumination': 0.99,
    'Name': ['["Clifton Hill Primary School, 185, Gold Street, Clifton Hill, City '
             'of Yarra, Victoria, 3068, Australia", "Clifton Hill Primary '
             'School", "185", "Australia", "City of Yarra", "Clifton Hill", '
             '"Victoria", "3068", "Gold Street"]'],
    'StartDateTime': '2019-06-16T14:00:00Z',
    'StartDateTimeLocal': '2019-06-17T00:00:00Z',
    'StreetViewImage': 'https://s3-us-west-1.amazonaws.com/unforgettable-dev-usw1/userhome/ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6/2019/06/08/streetview_full_4a28b74a-1608-448b-b8e4-7831b6d5f858.jpeg',
    'StreetViewThumbnail': 'https://s3-us-west-1.amazonaws.com/unforgettable-dev-usw1/userhome/ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6/2019/06/08/streetview_thumb_4a28b74a-1608-448b-b8e4-7831b6d5f858.jpeg',
    'Suggestions': [],
    'Temperature': 10.05,
    'UserId': '2f50d2d3ad5a5193945b47889a5c9a88f7a4ff70f1297973107911432a067b17',
    'UserImages': [],
    'Weather': 'Mostly Cloudy',
    'aws_profile': {'name': 'ume', 'region_name': 'us-west-1'},
    'hasAccelerometryDataFiles': True,
    'hasAccelerometryDataFilesItr': <Private.file_iterator.FileIterator object at 0x7f4d4ab4a550>,
    'hasAudioProcessedDataFiles': True,
    'hasAudioProcessedDataFilesItr': <Private.file_iterator.FileIterator object at 0x7f4d4ab4a080>,
    'hasGpsLocations': True,
    'id': 'ap-northeast-1:7fe8c602-eac8-4172-99ca-b3e07d27f1e6::ffc4284d-7aa3-415e-a3dd-b9125c556388',
    'latitude': -37.792988,
    'longitude': 144.988911,
    'type': '__App__'}