Google Fit exploration-2023-05-24-12:11:42.ipynb
Google fit 5/24/2023
Google Fit is an app for Android that stores your step count and other activity related metrics. It can also interface with other devices and get data from them as well. At OpenHumans, we have built an integration that queries the Google Fit API daily to get the step count, the distance, calories expended and active minutes, on a per minute granularity. Go here to connect your Google Fit data with your Open Humans account if you haven't done so already.
An important thing to know about Google Fit data is that a certain metric can be computed from different sources - for example, distance covered can be computed either from raw location data or be derived from the step counter. Google Fit stores the same metric multiple times, once per each source, and each source is tagged with the name of the sensor that produced the data. We will see concrete examples of this as we go along.
This notebook will show how to load the data that has been uploaded to Open Humans and ask questions across all your stored activity history.
The Google Fit data is stored on Open Humans in monthly json files. We are going to download all of them.
Make sure to set your own timezone. (List of timezone names)
We've written some functions to parse this json and load it into a pandas dataframe, accounting for data irregularities such as metrics and sources appearing and disappearing over time or metric/source pairs having no data. The end result is that we want to have a dataframe with columns step_count.delta.01, step_count.delta.02 etc, ie, suffix each different source for each metric with a number.
We now have a dataframe with columns for every metric/source pair and rows for every minute. We also returned a dictionary to map each column name to the associated source. The reason we do that is that the full source name tends to be a very large string. See below:
Below is what the dataframe looks like. We have four distinct metrics. active_minutes, calores.expended, distance.delta and step_count.delta, each with multiple data sources.
We can do a simple aggregation to get the total steps per day and plot it. The Google Fit app itself doesn't show exact numbers.
We can compare the distance from two different sources. In my dataset (yours can differ) distance.delta.01 uses the GPS and distance.delta.02 uses the step counter (which uses the gyroscope and other sensors from the phone). It is expected that those two sources would produce different results, and in fact, a more accurate distance metric could be computed using a Kalman filter that combines both data sources.
To check out the exact sources of your distance.delta columns, look into the source_mapper dictionary.
Now we're going to ask some more specific questions of the data, and answer them by using custom pandas functions.
The 10,000 steps per day recommended by the World Health Organization.
Google Fit is an app for Android that stores your step count and other activity related metrics. It can also interface with other devices and get data from them as well. At OpenHumans, we have built an integration that queries the Google Fit API daily to get the step count, the distance, calories expended and active minutes, on a per minute granularity. Go here to connect your Google Fit data with your Open Humans account if you haven't done so already.
An important thing to know about Google Fit data is that a certain metric can be computed from different sources - for example, distance covered can be computed either from raw location data or be derived from the step counter. Google Fit stores the same metric multiple times, once per each source, and each source is tagged with the name of the sensor that produced the data. We will see concrete examples of this as we go along.
This notebook will show how to load the data that has been uploaded to Open Humans and ask questions across all your stored activity history.
%matplotlib inline
import json
import pandas as pd
from datetime import datetime
import pytz
import sys
import requests
from functools import reduce
import os
The Google Fit data is stored on Open Humans in monthly json files. We are going to download all of them.
Make sure to set your own timezone. (List of timezone names)
token = os.environ.get('OH_ACCESS_TOKEN')
response = requests.get("https://www.openhumans.org/api/direct-sharing/project/exchange-member/?access_token={}".format(token))
# MONTH = '2018-12'
TIMEZONE = 'Europe/London'
user = json.loads(response.content.decode("utf-8"))
month_data = []
for dset in sorted(user['data'], key=lambda x:x['id']):
if 'Google Fit' in dset['metadata']['tags']: #and dset['metadata']['month'] == MONTH:
raw_data = requests.get(dset['download_url']).content
month_data.append(json.loads(raw_data.decode("utf-8")))
We've written some functions to parse this json and load it into a pandas dataframe, accounting for data irregularities such as metrics and sources appearing and disappearing over time or metric/source pairs having no data. The end result is that we want to have a dataframe with columns step_count.delta.01, step_count.delta.02 etc, ie, suffix each different source for each metric with a number.
def get_all_metrics(data):
return sorted(data['datasets'].keys())
def get_all_data_sources_for_metric(data, metric):
return sorted(data['datasets'][metric].keys())
def get_all_metrics_and_sources_pairs(data):
""" Get all data types and sources present in the data """
res = []
for metric in get_all_metrics(data):
sources = get_all_data_sources_for_metric(data, metric)
for src in sources:
res.append((metric, src))
return res
def get_dataframe(dataset, dt_timezone, col_name):
ts = []
for day in dataset.keys():
data = dataset[day].get('bucket', [])
for datum in data:
if datum['dataset'][0]['point'] == []:
value = 0
else:
try:
value = datum['dataset'][0]['point'][0]['value'][0]['intVal']
except:
value = datum['dataset'][0]['point'][0]['value'][0]['fpVal']
start_ms = datum['startTimeMillis']
start_sec = int(start_ms) / 1000
dt = datetime.utcfromtimestamp(start_sec)
dt = pytz.timezone('UTC').localize(dt)
dt_local = dt.astimezone(pytz.timezone(dt_timezone))
ts.append((dt_local, value))
if len(ts) == 0:
return None
df = pd.DataFrame(ts)
df.columns = ['time', col_name]
df = df.set_index('time')
return df
def generate_unique_column_name(metric, existing_names):
i = 1
while True:
# turn com.google.steps.delta to steps.delta
metric_formatted = '.'.join(metric.split('.')[2:])
name = metric_formatted + '.' + str(i).zfill(2)
if name in existing_names:
i+=1
continue
else:
return name
def get_dataframe_with_metrics_and_sources(dts_pairs, col_names, timezone, data):
dfs = []
for (metric, dsource), col_name in zip(dts_pairs, col_names):
dts_data = data['datasets'].get(metric, {}).get(dsource, {})
df = get_dataframe(dts_data, timezone, col_name=col_name)
if df is not None:
dfs.append(df)
if len(dfs) > 0:
df = reduce(lambda acc, df: acc.join(df, how='outer'), dfs)
df = df.fillna(0)
else:
df = None
return df
def get_renames(columns):
renames = {}
columns = sorted(columns)
new_columns = []
for col in columns:
new_col = get_renamed_column(col, new_columns)
new_columns.append(new_col)
renames[col] = new_col
return renames
def get_renamed_column(col, new_columns):
num = int(col.split('.')[-1])
if num == 1:
return col
while (num > 1):
current_possible_name = '.'.join(col.split('.')[:-1])+ '.' + str(num).zfill(2)
previous_possible_name = '.'.join(col.split('.')[:-1])+ '.' + str(num-1).zfill(2)
if previous_possible_name not in new_columns:
num = num - 1
else:
return current_possible_name
return previous_possible_name
def googlefit_jsons_to_df(data, timezone):
dts_pairs = set()
mapping = {}
dfs = []
for month_data in data:
current_dts_pairs = set(get_all_metrics_and_sources_pairs(month_data))
dts_pairs = dts_pairs | current_dts_pairs
dts_pairs = sorted(list(dts_pairs))
col_names = []
for dtype, dsource in dts_pairs:
col_name = generate_unique_column_name(dtype, col_names)
col_names.append(col_name)
mapping[col_name] = dsource
for month_data in data:
df = get_dataframe_with_metrics_and_sources(dts_pairs, col_names, timezone, month_data)
if df is not None:
dfs.append(df)
if len(dfs) == 0:
return None, mapping
result = pd.concat(dfs).fillna(0)
column_renames = get_renames(result.columns)
for col, source in mapping.copy().items():
new_col = column_renames.get(col)
if new_col is None:
continue
mapping[new_col] = source
return result.rename(columns=column_renames), mapping
df, source_mapper = googlefit_jsons_to_df(month_data, TIMEZONE)
We now have a dataframe with columns for every metric/source pair and rows for every minute. We also returned a dictionary to map each column name to the associated source. The reason we do that is that the full source name tends to be a very large string. See below:
print("The data source for step_count.delta.01 is {}".format(source_mapper['step_count.delta.01']))
print("The data source for step_count.delta.02 is {}".format(source_mapper['step_count.delta.02']))
Below is what the dataframe looks like. We have four distinct metrics. active_minutes, calores.expended, distance.delta and step_count.delta, each with multiple data sources.
df.head()
We can do a simple aggregation to get the total steps per day and plot it. The Google Fit app itself doesn't show exact numbers.
df['step_count.delta.01'].resample('D').sum()
ax = df[['step_count.delta.01', 'step_count.delta.02']].resample('D').sum().plot()
ax.grid(True, which='minor', axis='x')
ax.grid(True, which='major', axis='x')
We can compare the distance from two different sources. In my dataset (yours can differ) distance.delta.01 uses the GPS and distance.delta.02 uses the step counter (which uses the gyroscope and other sensors from the phone). It is expected that those two sources would produce different results, and in fact, a more accurate distance metric could be computed using a Kalman filter that combines both data sources.
To check out the exact sources of your distance.delta columns, look into the source_mapper dictionary.
ax = df[['distance.delta.01', 'distance.delta.02']].resample('D').sum().plot()
ax.grid(True, which='minor', axis='x')
ax.grid(True, which='major', axis='x')
print("The data source for distance.delta.01 is {}".format(source_mapper['distance.delta.01']))
print("The data source for distance.delta.02 is {}".format(source_mapper['distance.delta.02']))
Now we're going to ask some more specific questions of the data, and answer them by using custom pandas functions.
weekdays = df[df.index.dayofweek < 5]
weekends = df[df.index.dayofweek >= 5]
avg_weekdays = weekdays['active_minutes.02'].resample('D').sum().mean()
avg_weekends = weekends['active_minutes.02'].resample('D').sum().mean()
print("Average active minutes per day on weekdays: {}".format(int(avg_weekdays)))
print("Average active minutes per day on weekends: {}".format(int(avg_weekends)))
print("Day with most steps walked ({}) is {}".format(
int(df['step_count.delta.01'].resample('D').sum().max()),
df['step_count.delta.01'].resample('D').sum().idxmax().strftime("%Y-%m-%d"),
))
print("Day with most active minutes ({}) is {}".format(
int(df['active_minutes.02'].resample('D').sum().max()),
df['active_minutes.02'].resample('D').sum().idxmax().strftime("%Y-%m-%d"),
))
df['week_no_year'] = df.index.map(lambda x: str(x.date().isocalendar()[0]) + "_" + str(x.date().isocalendar()[1]))
print("Day with most steps per week")
print(df.groupby('week_no_year')['step_count.delta.01'].agg(
lambda x: (x.resample('D').sum().idxmax().strftime("%a %d/%m"), int(x.resample('D').sum().max()))
).sort_index())
The 10,000 steps per day recommended by the World Health Organization.
GOAL = 10000
daily_steps = df['step_count.delta.01'].resample('D').sum()
print("Reached {} steps for {} days out of {}".format(GOAL, (daily_steps>GOAL).sum(), len(daily_steps)))