Details for dashboard-oura-spotify.ipynb

Published by gedankenstuecke

Description

This notebook is an attempt at making an interactive dashboard to correlate many of the different variables tracked by either the Oura Ring or Spotify through the use of the plotly library along with interactive widges for Jupyter notebooks.

Ironically, this dashboard currently doesn't work well in the voila dashboard view, which means you should select Edit & Run in the Exploratory to render it in the regular Jupyter interface! Select Cell and then Run All from the menu in the Jupyter interface (This is because the plotly library is currently not installed by default and takes a while to install).

0

Tags & Data Sources

plotly dashboard interactive Oura Connect Spotify integration

Comments

Please log in to comment.

Notebook
Last updated 8 months, 2 weeks ago

Interactive Dashboards for Spotify and Oura Ring Data

If you want to run this notebook and run into problems or have questions: Reach out to Bastian on Twitter or Slack

This notebook is an attempt at making an interactive dashboard to correlate many of the different variables tracked by either the Oura Ring or Spotify through the use of the plotly library along with interactive widges for Jupyter notebooks.

Ironically, this dashboard currently doesn't work well in the voila dashboard view, which means you should select Edit & Run in the Exploratory to render it in the regular Jupyter interface! Select Cell and then Run All from the menu in the Jupyter interface (This is because the plotly library is currently not installed by default and takes a while to install).

For this notebook to work you should have at least one of those two data sources connected to your Open Humans account:

With that out of the way, we'll get started to load the libraries and data and process it. Feel free to skip to the very end of the notebook to see the plots!

Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.8.2
  latest version: 4.9.2

Please update conda by running

    $ conda update -n base conda



# All requested packages already installed.

Oura Ring data exploration

Now we're getting to the first interactive plot. Below is the visualization and settings for the Oura data. You can select which data should be shown on the X & Y axis, along with the visualizations for the data distribution that you want to be shown on the margins. You can also opt in to see a linear or local fitting of the data, depending on your preference (this works only for numeric data, not for dates yet).

You can hover over each data point, the marginal plots etc and also zoom in/out for your regions of interest.

Visualizing the Spotify data!

Now we're getting to the second interactive plot. Below is the visualization and settings for the Spotify data. You can select which data should be shown on the X & Y axis, along with the visualizations for the data distribution that you want to be shown on the margins. You can also opt in to see a linear or local fitting of the data, depending on your preference (this works only for numeric data, not for dates yet).

You can hover over each data point, the marginal plots etc and also zoom in/out for your regions of interest.

Have fun exploring your data, and join #self-research in the OH Slack if you found something interesting you want to share!

Notebook
Last updated 8 months, 2 weeks ago

Interactive Dashboards for Spotify and Oura Ring Data

If you want to run this notebook and run into problems or have questions: Reach out to Bastian on Twitter or Slack

This notebook is an attempt at making an interactive dashboard to correlate many of the different variables tracked by either the Oura Ring or Spotify through the use of the plotly library along with interactive widges for Jupyter notebooks.

Ironically, this dashboard currently doesn't work well in the voila dashboard view, which means you should select Edit & Run in the Exploratory to render it in the regular Jupyter interface! Select Cell and then Run All from the menu in the Jupyter interface (This is because the plotly library is currently not installed by default and takes a while to install).

For this notebook to work you should have at least one of those two data sources connected to your Open Humans account:

With that out of the way, we'll get started to load the libraries and data and process it. Feel free to skip to the very end of the notebook to see the plots!

In [1]:
import sys
!conda install --yes --prefix {sys.prefix} plotly

# Standard Data Science Helpers
import numpy as np
import pandas as pd
import scipy
import plotly.graph_objects as go
import plotly.express as px
from IPython.display import Image, display, HTML
import ipywidgets as widgets
from ipywidgets import interact, interact_manual
Collecting package metadata (current_repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.8.2
  latest version: 4.9.2

Please update conda by running

    $ conda update -n base conda



# All requested packages already installed.

In [2]:
### GET DATA FOR RESCUETIME, OURA AND SPOTIFY

from ohapi import api
import os
import requests
import tempfile
import json 
import pandas as pd
from datetime import datetime

oura_present = ""
rescuetime_present = ""
spotify_present = ""

df_moment = ""
dataframe_oura_full = ""
rt_df_full = ""
df_spotify = ""

user_details = api.exchange_oauth2_member(os.environ.get('OH_ACCESS_TOKEN'))
for i in user_details['data']:
    if i['source'] == 'direct-sharing-184' and i['basename'] == 'oura-data.json':
        oura = json.loads(requests.get(i['download_url']).content)
        oura_present = "True"
    if i['source'] == 'direct-sharing-176' and i['basename'] == 'spotify-listening-archive.json':
        sp_songs = requests.get(i['download_url'])
        sp_data = json.loads(sp_songs.content)
        spotify_present = "True"
    if i['source'] == 'direct-sharing-176' and i['basename'] == 'spotify-track-metadata.json':
        sp_meta = requests.get(i['download_url'])
        sp_metadata = json.loads(sp_meta.content)
        
### PARSERS FOR OURA, SPOTIFY & RESCUETIME
def read_oura(oura):

    dates = []
    values = []
    value_type = []

    for sdate in oura['sleep']:
        dates.append(sdate['summary_date'])
        values.append(sdate['score'])
        value_type.append('sleep: score')
        dates.append(sdate['summary_date'])
        values.append(sdate['total'])
        value_type.append('sleep: sleep sum')
        dates.append(sdate['summary_date'])
        values.append(sdate['hr_lowest'])
        value_type.append('sleep: hr_lowest')
        dates.append(sdate['summary_date'])
        values.append(sdate['hr_average'])
        value_type.append('sleep: hr_average')
        dates.append(sdate['summary_date'])
        values.append(sdate['rmssd'])
        value_type.append('sleep: rmssd')
        dates.append(sdate['summary_date'])
        values.append(sdate['rem'])
        value_type.append('sleep: rem')
        dates.append(sdate['summary_date'])
        values.append(sdate['restless'])
        value_type.append('sleep: restless')
        dates.append(sdate['summary_date'])
        values.append(sdate['awake'])
        value_type.append('sleep: awake')
        dates.append(sdate['summary_date'])
        values.append(sdate['light'])
        value_type.append('sleep: light')
        dates.append(sdate['summary_date'])
        values.append(sdate['deep'])
        value_type.append('sleep: deep')
        dates.append(sdate['summary_date'])
        values.append(sdate['breath_average'])
        value_type.append('sleep: breath_average')
        dates.append(sdate['summary_date'])
        values.append(sdate['onset_latency'])
        value_type.append('sleep: onset_latency')
        dates.append(sdate['summary_date'])
        values.append(sdate['temperature_delta'])
        value_type.append('sleep: temperature_delta')

        
        
    for sdate in oura['activity']:
        dates.append(sdate['summary_date'])
        values.append(sdate['score'])
        value_type.append('activity: score')
        dates.append(sdate['summary_date'])
        values.append(sdate['steps'])
        value_type.append('activity: steps')
        dates.append(sdate['summary_date'])
        values.append(sdate['cal_active'])
        value_type.append('activity: cal active')
        dates.append(sdate['summary_date'])
        values.append(sdate['cal_total'])
        value_type.append('activity: cal total')
        dates.append(sdate['summary_date'])
        values.append(sdate['daily_movement'])
        value_type.append('activity: daily movement')
        dates.append(sdate['summary_date'])
        values.append(sdate['high'])
        value_type.append('activity: high')
        dates.append(sdate['summary_date'])
        values.append(sdate['low'])
        value_type.append('activity: low')
        dates.append(sdate['summary_date'])
        values.append(sdate['inactive'])
        value_type.append('activity: inactive')
        dates.append(sdate['summary_date'])
        values.append(sdate['medium'])
        value_type.append('activity: medium')
        dates.append(sdate['summary_date'])
        values.append(sdate['rest'])
        value_type.append('activity: rest')
        dates.append(sdate['summary_date'])
        values.append(sdate['inactivity_alerts'])
        value_type.append('activity: inactivity alerts')
        dates.append(sdate['summary_date'])
        values.append(sdate['steps'])
        value_type.append('activity: total')

        
    for sdate in oura['readiness']:
        dates.append(sdate['summary_date'])
        values.append(sdate['score'])
        value_type.append('readiness score')


    dataframe = pd.DataFrame(
        data = {
            'date': dates,
            'value': values,
            'type': value_type
        }
    )
    return dataframe

def read_rescuetime(rescuetime_data):
    date = []
    time_spent_seconds = []
    activity = []
    category = []
    productivity = []
    for element in rescuetime_data['rows']:
        date.append(element[0])
        time_spent_seconds.append(element[1])
        activity.append(element[3])
        category.append(element[4])
        productivity.append(element[5])
    date = [datetime.strptime(dt,"%Y-%m-%dT%H:%M:%S") for dt in date]

    rt_df = pd.DataFrame(data={
        'date': date,
        'time_spent_seconds': time_spent_seconds,
        'activity': activity,
        'category': category,
        'productivity': productivity
    })
    return rt_df

def parse_timestamp(lst):
    timestamps = []
    for item in lst:
        try:
            timestamp = datetime.strptime(
                            item,
                            '%Y-%m-%dT%H:%M:%S.%fZ')
        except ValueError:
            timestamp = datetime.strptime(
                    item,
                    '%Y-%m-%dT%H:%M:%SZ')
        timestamps.append(timestamp)
    return timestamps

def read_spotify(sp_data, sp_metadata):
    track_title = []
    artist_name = []
    album_name = []
    played_at = []
    popularity = []
    duration_ms = []
    explicit = []
    track_id = []

    danceability = []
    energy = [] 
    key = [] 
    loudness = [] 
    mode = []
    speechiness = []
    acousticness = []
    instrumentalness = []
    liveness = []
    valence = []
    tempo = []

    ['danceability', 'energy', 'key', 'loudness', 'mode', 'speechiness', 'acousticness', 'instrumentalness', 'liveness', 'valence','tempo']


    key_translation = {
        '0': "C",
        "1": "C#",
        "2": "D",
        "3": "D#",
        "4": "E", 
        "5": "F",
        "6": "F#",
        "7": "G",
        "8": "G#",
        "9": "A",
        "10": "A#",
        "11": "B"
    }

    mode_translation = {'1':'major', '0': 'minor'}

    for sp in sp_data:
        if sp['track']['id'] in sp_metadata.keys():
            track_title.append(sp['track']['name'])
            artist_name.append(sp['track']['artists'][0]['name'])
            album_name.append(sp['track']['album']['name'])
            played_at.append(sp['played_at'])
            popularity.append(sp['track']['popularity'])
            duration_ms.append(sp['track']['duration_ms'])
            explicit.append(sp['track']['explicit'])
            track_id.append(sp['track']['id'])
            danceability.append(sp_metadata[sp['track']['id']]['danceability'])
            energy.append(sp_metadata[sp['track']['id']]['energy'])
            key.append(key_translation[str(sp_metadata[sp['track']['id']]['key'])])
            loudness.append(sp_metadata[sp['track']['id']]['loudness'])
            mode.append(mode_translation[str(sp_metadata[sp['track']['id']]['mode'])])
            speechiness.append(sp_metadata[sp['track']['id']]['speechiness'])
            acousticness.append(sp_metadata[sp['track']['id']]['acousticness'])
            instrumentalness.append(sp_metadata[sp['track']['id']]['instrumentalness'])
            liveness.append(sp_metadata[sp['track']['id']]['liveness'])
            valence.append(sp_metadata[sp['track']['id']]['valence'])
            tempo.append(sp_metadata[sp['track']['id']]['tempo'])



    played_at = parse_timestamp(played_at)

    dataframe = pd.DataFrame(data={
        'track_id': track_id,
        'track': track_title,
        'artist': artist_name,
        'album': album_name,
        'popularity': popularity,
        'duration_ms': duration_ms,
        'explicit': explicit,
        'played_at': played_at,
        'danceability': danceability,
        'energy': energy,
        'key': key,
        'loudness': loudness,
        'mode': mode,
        'speechiness': speechiness,
        'acousticness': acousticness,
        'instrumentalness': instrumentalness,
        'liveness': liveness,
        'valence': valence,
        'tempo': tempo

    })
    dataframe = dataframe
    return dataframe

Oura Ring data exploration

Now we're getting to the first interactive plot. Below is the visualization and settings for the Oura data. You can select which data should be shown on the X & Y axis, along with the visualizations for the data distribution that you want to be shown on the margins. You can also opt in to see a linear or local fitting of the data, depending on your preference (this works only for numeric data, not for dates yet).

You can hover over each data point, the marginal plots etc and also zoom in/out for your regions of interest.

In [3]:
### CREATE DATAFRAMES
if oura_present:
    dataframe_oura_full = read_oura(oura)
    oura_pivot = dataframe_oura_full.pivot_table(index='date',columns='type',values='value')
    oura_pivot = oura_pivot.reset_index()
    
    
    def plot_oura(x, y,marginal,fit):
        marginal_x = None
        marginal_y = None
        trendline = None
        if marginal != 'none':        
            if oura_pivot[x].dtypes == 'float64':
                marginal_x = marginal
            if oura_pivot[y].dtypes == 'float64':
                marginal_y = marginal
        if marginal_x and marginal_y:
            if fit == 'linear (ols)':
                trendline = 'ols'
            if fit == 'loess':
                trendline = 'lowess'
        fig = px.scatter(oura_pivot, x=x, y=y, hover_data=[x,y,'date'], marginal_y=marginal_y,
                marginal_x=marginal_x, trendline=trendline, template="simple_white")
        fig.show()


    _ = widgets.interact(
        plot_oura, 
        x=oura_pivot.columns,
        y=oura_pivot.columns,
        marginal=['violin','box','none'],
        fit=['linear (ols)','loess','none']
    )

Visualizing the Spotify data!

Now we're getting to the second interactive plot. Below is the visualization and settings for the Spotify data. You can select which data should be shown on the X & Y axis, along with the visualizations for the data distribution that you want to be shown on the margins. You can also opt in to see a linear or local fitting of the data, depending on your preference (this works only for numeric data, not for dates yet).

You can hover over each data point, the marginal plots etc and also zoom in/out for your regions of interest.

In [4]:
if spotify_present:
    df_spotify = read_spotify(sp_data, sp_metadata)
    def plot_spotify(x, y,marginal,fit):
        marginal_x = None
        marginal_y = None
        trendline = None
        if marginal != 'none':        
            if df_spotify[x].dtypes == 'float64' or df_spotify[x].dtypes == 'int64':
                marginal_x = marginal
            if df_spotify[y].dtypes == 'float64' or df_spotify[x].dtypes == 'int64':
                marginal_y = marginal
        if marginal_x and marginal_y:
            if fit == 'linear (ols)':
                trendline = 'ols'
            if fit == 'loess':
                trendline = 'lowess'
        fig = px.scatter(df_spotify, x=x, y=y,marginal_y=marginal_y, hover_data=[x,y,'played_at'],
                marginal_x=marginal_x, trendline=trendline, template="simple_white")
        fig.show()


    _ = widgets.interact(
        plot_spotify, 
        x=['popularity','duration_ms','played_at','danceability','energy','loudness'],
        y=['popularity','duration_ms','played_at','danceability','energy','loudness'],
        marginal=['violin','box','none'],
        fit=['linear (ols)','loess','none']
    )

Have fun exploring your data, and join #self-research in the OH Slack if you found something interesting you want to share!