Timestamp_16 support

[buma]

Currently in FIT we have timestamps which are 32bit and timestamp_16 which are 16bit difference from previous timestamp and are currently not read into nice values.

I wrote generator function based on ParseVivosmartHR. Here seems to be offical function how to do it (in Java):

mesgTimestamp += ( timestamp_16 - ( mesgTimestamp & 0xFFFF ) ) & 0xFFFF; 

It works but it definitely isn't written as it should, but I don't know fitparse enough to improve it.

#Input is what we get in FitFile.get_messages
def fix_times(messages):                                                        
    timestamp=None                                                              
    timestamp_16=None                                                           
    real_time=0                                                                 
    UTC_REFERENCE = 631065600  # timestamp for UTC 00:00 Dec 31 1989            
    for message in messages:                                                    
        #print (message)                                                        
        #message["full_timestamp"]="TIMESTAMP"                                  
        field = message.get("timestamp")                                        
        if field is not None:                                                   
            timestamp = field.raw_value                                         
            #I'm not sure if this is correct to set this to None. Without it records with just timestamp doesn't have same new timestamp anymore which could be a bug or it should be that way                                          
            timestamp_16 = None                                                 
        tm_16 = message.get("timestamp_16")                                     
        if tm_16 is not None:                                                   
            timestamp_16 = tm_16.raw_value                                      
        #print(timestamp, timestamp_16)                                         
        if timestamp_16 is None:                                                
            ts_value = timestamp                                                
        else:                                                                   
            new_time = int( timestamp/2**16) * 2**16 + timestamp_16             
            if new_time >= real_time:                                           
                real_time = new_time                                            
            else:                                                               
                real_time = new_time + 2**16                                    
            ts_value = real_time                                                
        if ts_value is not None and ts_value >= 0x10000000:                     
            value = datetime.datetime.utcfromtimestamp(UTC_REFERENCE            
                    + ts_value)                                                 
        else:                                                                   
            value = None                                                        
        message.fields.append(                                                  
                FieldData(                                                      
                    field_def=None,                                             
                    field=FIELD_TYPE_TIMESTAMP_1,                               
                    parent_field=None,                                          
                    value=value,                                                
                    raw_value=ts_value                                          
                    ))                                                          
        yield message

[pR0Ps]

As an FYI for testing this there is already a file that has these timestamp_16 fields in the repo: tests/files/MonitoringFile.fit

[danpatdav]

Any progress on this issue? I am struggling to make sense of heartrate records which utilize timestamp_16.

[brandonStell]

Hi @buma, I seem to be doing something wrong. How should fix_times be called?

for message in fix_times(fitfile.get_messages()):
    ts =  message.get_value('timestamp_16')
    if ts:
        print ts
        
Traceback (most recent call last):
  File "/home/bs/.local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2878, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-64-a1a3e382974a>", line 1, in <module>
    for message in fix_times(fitfile.get_messages()):
  File "<ipython-input-25-b548c6307590>", line 33, in fix_times
    FieldData(
NameError: global name 'FieldData' is not defined

[danpatdav]

Happy to test any updated code.

[francescolosterzo]

Hello! Happy to see this discussed here :)
I am struggling with the same problem, and for me the formula in the top post doesn't work! Following the instructions from here, I tried to use that formula by hand in order to see if things made sense.

I downloaded my heart rate data recorded on May 1st 2019 from the Garmin Connect web page, i.e. the first data point is at 12AM.

Looping through the FitFile.get_messages(), the first heart-rate-related entry I find is this, including the message right before it:

monitoring
 * activity_type: sedentary
 * current_activity_type_intensity: (8,)
 * intensity: 0
 * timestamp: 2019-04-30 22:00:00
monitoring
 * heart_rate: 72
 * timestamp_16: 31132

Following the instructions linked above, I set

dt = datetime.datetime(2019,4,30,22)
mesgTimestamp = datetime.datetime.timestamp(dt)
timestamp_16 = 31132

but as a result of the above formula I get

print('result:', datetime.datetime.fromtimestamp(mesgTimestamp))
>> result: 2019-05-01 12:49:00

which is wrong!

Has anyone found a fix to this?

[francescolosterzo]

I tried to look at things in detail going step by step in the above formula:

data:
mesgTimestamp: 2019-04-30 22:00:00 -> 1556659800 :   0b1011100110010001011111001011000
timestamp_16: 31132                              :                   0b111100110011100
start:
step1: mesgTimestamp & 0xFFFF                    :                  0b1011111001011000
step2: timestamp_16 - step1                      :                  -0b100010010111100
step3: step2 & 0xFFFF                            :                  0b1011101101000100
step4: mesgTimestamp + step3                     :   0b1011100110010010111100110011100 -> 1556707740 -> 2019-05-01 12:49:00

and, as before the result is wrong by more than 12 hours.
(note that step2 is negative!)

Also, the instructions say

essentially swaps the lower 16 bits of the timestamp field with the timestamp_16 field

but the final result is not a 32-bit variable in which the first 16 are from the original mesgTimestamp and the last 16 are from timestamp_16, some bits are flipped!

On the other hand I tried to build it by hand, and I get the final binary variable as:

0b1011100110010001111100110011100

which then translates to this datetime:

datetime.datetime(2019, 5, 1, 3, 42, 52)

which, again, is wrong :\ but different from before!

[tuxinaut]

Does this maybe help?

adjust_time.R

[francescolosterzo]

thanks @tuxinaut ! I tried it but I still get a sizeable difference, i.e.: datetime.datetime(2019, 4, 30, 18, 36, 44)

On the other hand I browsed the details of the code and I see that the UTC offset shift due to Garmin's different starting time is applied to timestamp, but I cannot find anything similar for timestamp_16.

Following this hypothesis I tried to work with raw timestamp* values (I guess they are the values before this UTC offset correction) and here is what I get:

# values taken from full printouts
timestamp_raw = 925596000
timestamp_16_raw = 31132

out = timestamp_raw
out += ( timestamp_16 - ( out & 0xFFFF ) ) & 0xFFFF

datetime.datetime.fromtimestamp(out + utc_offset)

The result is datetime.datetime(2019, 4, 30, 23, 1)

[here is the full notebook, just search for "925596000"]

This is the closest I got so far to the right result, and it is shifted only by one hour (the minute might be due to the granularity of the data) which might now be a timezone related issue.

What do you think @dtcooper ?

[tuxinaut]

@francescolosterzo really cool work 👍 and yes I'm pretty sure that's timezone related (take a look on the code below)

>>> timestamp_16 = 31132
>>> tz = pytz.timezone('UTC')
>>> utc_offset = int(datetime.datetime.timestamp(datetime.datetime(1989,12,31,tzinfo=tz)))
>>> timestamp = int(datetime.datetime.timestamp(datetime.datetime(2019,4,30,22,0,tzinfo=tz)))
>>> timestamp_raw = timestamp - utc_offset
>>> out = timestamp_raw
>>> out += ( timestamp_16 - ( out & 0xFFFF ) ) & 0xFFFF
>>> datetime.datetime.fromtimestamp(out + utc_offset,tz=tz)
datetime.datetime(2019, 4, 30, 22, 1, tzinfo=<UTC>)

[mszczepanczyk]

This is what I came up with after reading this thread. The generated timestamps and bpms are in line with the data in Garming Connect.

import fitparse

from datetime import datetime, timezone
from glob import glob

garmin_epoch = int(datetime.timestamp(datetime(1989, 12, 31, tzinfo=timezone.utc)))

files = sorted(glob("2023-10-25/*WELLNESS*.fit"))
for f in files:
    fitfile = fitparse.FitFile(f)
    last_timestamp = None
    for m in fitfile.get_messages("monitoring"):
        last_timestamp = m.get_raw_value("timestamp") or last_timestamp
        heart_rate = m.get_raw_value("heart_rate")
        timestamp16 = m.get_raw_value("timestamp_16")
        if last_timestamp and heart_rate and timestamp16:
            timestamp = last_timestamp
            timestamp += (timestamp16 - (last_timestamp & 0xFFFF)) & 0xFFFF
			date = datetime.fromtimestamp(garmin_epoch + timestamp, tz=timezone.utc)
            print(date.astimezone().strftime("%Y-%m-%dT%H:%M:%S %Z"), heart_rate)

2023-10-25T00:01:00 CEST 75
2023-10-25T00:02:00 CEST 72
2023-10-25T00:03:00 CEST 73
...