Make the auto-import more flexible

[evensolberg]

Right now it's hardcoded to look in a /Activity subdirectory in the specified Garmin folder. I import my files from other devices (Wahoo) using other mechanisms, and I'm not using a /Activity folder. It would be nice if it could be optional to NOT look in that folder.

[fgebhart]

Hi @evensolberg, thanks for opening up that issue.

I think it would make sense to improve the flexibility of the import mechanism. Up until now I simply assumed all devices had an Activity directory in which it stores its fit files. In case the name of your "activities" folder is different this won't work of course.

One potential solution to this could be to make the folder name in which the activities are stores configurable via the settings page. This would be similar to what was done in #91 (what was changed again later on).
In order to better understand the situation with e.g. Wahoo devices, it would be great if you could share the full path in which your activity fit files are stored (once the file system of your device is mounted to your system).

For my Garmin device this would for example be

/run/user/1000/gvfs/mtp\:host\=091e_4b48_0000c4fa0516/Primary/GARMIN/Activity/<fit_file>.fit

Do you think this approach could work and solve your issue?

[evensolberg]

To be honest, I don't even know where the files are stored. I normally email them to myself and then dump them out into a directory, since I use them in multiple applications. So yes, being able to just point to an arbitrary directory would work nicely for me.

I tried copying the files to an /Activity folder, but somehow "Reimport Files" doesn't seem to work. Is there any way to see what's going on?

[fgebhart]

Your workflow sounds to me like you do have your activity files already stored on your local system (opposed to on your watch/device). In that case the settings option called "Path to Traces Directory" should do exactly what you are looking for.

For the sake of clarity, let me explain the difference:

• Path to Garmin Device: Path to a directory in which your device/watch will be mounted in, in order to copy the fit files from the device onto the system workoutizer is running on. This process is called fit collector within the code base. The fit collector searches for a directory called Activity/ in order to copy the activity files from there. That is required since typically the sport devices have many folders with multiple different purposes, many of them containing fit files - in which workoutizer is not interested in. The directory specified is the source location for copying the fit files, target location is the "Path to Traces Directory".
• Path to Traces Directory: Path to a directory on the same system on which workoutizer is running, e.g. your laptop (sometimes a Raspberry Pi, ...). The directory specified in this field could contain fit and gpx files in an arbitrary hierarchy. Workoutizer will check the folder for new activity files whenever it is started and will realize when a new file is added. New/unknown files are automatically imported. This processes is called File Importer in the code base. The path specified here could be any arbitrary path and does not require an Activity/ directory.

I suggest you give this option a try for importing your activity fit files into workoutizer.

Note, this holds true for the current version 0.17.1 - things might change in upcoming versions 😉

[...] but somehow "Reimport Files" doesn't seem to work. Is there any way to see what's going on?

For debugging you could run workoutizer in debug mode, by setting the following environment variable prior to running it:

export WKZ_LOG_LEVEL="DEBUG"

You should be able to see something like

04-25 19:37:33 - DEBUG - wkz.apps - started watchdog for incoming activity files in /workspaces/workoutizer/tracks

Where /workspaces/workoutizer/tracks should be the folder containing your activity files.

Let me know whether this is of any help or if I totally got your workflow wrong 😉

[evensolberg]

I will give this a try, thanks! I didn't realize that's what the other option was for. :)

It would be nice to have some kind of feedback in the user interface during the import, though, since I have 100+ files in the directory, and no way of knowing if anything is happening.

[fgebhart]

Yea, you are right. The only feedback one could currently see, is by looking at the command line - which is of course not a good user experience... I created #123 for this purpose and will think about how this could be solved. Ideas or contributions on this are welcome ;)

Any news on the usage of Path to Traces Directory?

[evensolberg]

Nothing seems to be happening with the Path to Traces Directory. I changed it, and clicked the Reimport button, and nothing happens. Some feedback mechanism would be nice. ;-)

As for the feedback in the UI, I'd love to see progression and maybe a tail of the last few log messages. If the logs could be output to a file, that would be handy too.

[fgebhart]

Ok, I get it... some feedback would be nice - agree on that :D (#123)

After changing the Path to Traces Directory you would need to re-run workoutizer to have the change become effective. For the initial setup/configuration and debugging I would recommend to have a closer look into the logging output in the terminal. Increasing verbosity by setting export WKZ_LOG_LEVEL="DEBUG" as mentioned above should give:

04-27 19:36:45 - DEBUG - wkz.file_importer - found 10 files in trace dir: /workspaces/workoutizer/tracks

[evensolberg]

Okay, I was finally able to get to this. Apologies for the delay. Here's what I'm seeing in the log. I went into the settings and clicked "Reimport Files" at 17:27:16.

05-02 17:26:37 - INFO - wkz.apps - using workoutizer home at /home/evensolberg/.wkz
05-02 17:26:37 - DEBUG - wkz.file_importer - triggered file importer
05-02 17:26:37 - DEBUG - wkz.file_importer - found 111 files in trace dir: /mnt/home/Drive/Drive/Data Analysis/Wahoo/Logs/FIT
05-02 17:26:37 - DEBUG - wkz.file_importer - importing file /mnt/home/Drive/Drive/Data Analysis/Wahoo/Logs/FIT/2019-04-12-140638-FITNESS 416C-2-12.fit ...
05-02 17:26:37 - DEBUG - wkz.file_importer - parsing FIT file ...
05-02 17:26:38 - INFO - django.utils.autoreload - Watching for file changes with StatReloader
Performing system checks...

System check identified no issues (0 silenced).
May 02, 2021 - 17:26:38
Django version 3.2, using settings 'workoutizer.settings'
Starting development server at http://192.168.1.16:8000/
Quit the server with CONTROL-C.
05-02 17:27:05 - WARNING - wkz.views - no activities found...
05-02 17:27:05 - INFO - django.server - "GET / HTTP/1.1" 200 23289
05-02 17:27:05 - DEBUG - wkz.views - fetching activity data for table page 0
05-02 17:27:05 - DEBUG - wkz.views - reached end of the table
05-02 17:27:05 - INFO - django.server - "GET /activities_page/0 HTTP/1.1" 200 4
05-02 17:27:05 - INFO - django.server - "GET /static/img/favicon.ico HTTP/1.1" 200 311
05-02 17:27:16 - INFO - django.server - "GET /settings/ HTTP/1.1" 200 17963
05-02 17:27:18 - INFO - django.server - "GET /settings/reimport HTTP/1.1" 302 0
05-02 17:27:18 - DEBUG - wkz.file_importer - triggered file importer
05-02 17:27:19 - DEBUG - wkz.file_importer - found 111 files in trace dir: /mnt/home/Drive/Drive/Data Analysis/Wahoo/Logs/FIT
05-02 17:27:19 - INFO - django.server - "GET /settings/ HTTP/1.1" 200 18201
05-02 17:27:19 - DEBUG - wkz.file_importer - importing file /mnt/home/Drive/Drive/Data Analysis/Wahoo/Logs/FIT/2019-04-12-140638-FITNESS 416C-2-12.fit ...
05-02 17:27:19 - DEBUG - wkz.file_importer - parsing FIT file ...
Process Process-1:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/evensolberg/.local/lib/python3.8/site-packages/wkz/file_importer.py", line 57, in reimport_activity_files
    _run_file_importer(models, importing_demo_data=False, reimporting=True)
  File "/home/evensolberg/.local/lib/python3.8/site-packages/wkz/file_importer.py", line 73, in _run_file_importer
    _run_parser(models, trace_files, importing_demo_data, reimporting)
  File "/home/evensolberg/.local/lib/python3.8/site-packages/wkz/file_importer.py", line 87, in _run_parser
    activity = _parse_and_save_to_model(
  File "/home/evensolberg/.local/lib/python3.8/site-packages/wkz/file_importer.py", line 129, in _parse_and_save_to_model
    parser = _parse_data(trace_file)
  File "/home/evensolberg/.local/lib/python3.8/site-packages/wkz/file_importer.py", line 356, in _parse_data
    parser = FITParser(path_to_file=file)
  File "/home/evensolberg/.local/lib/python3.8/site-packages/wkz/file_helper/fit_parser.py", line 20, in __init__
    super(FITParser, self).__init__(path_to_file)
  File "/home/evensolberg/.local/lib/python3.8/site-packages/wkz/file_helper/parser.py", line 83, in __init__
    self._parse_records()
  File "/home/evensolberg/.local/lib/python3.8/site-packages/wkz/file_helper/fit_parser.py", line 30, in _parse_records
    for record in self.fit.get_messages():
  File "/home/evensolberg/.local/lib/python3.8/site-packages/fitparse/base.py", line 434, in get_messages
    message = self._parse_message()
  File "/home/evensolberg/.local/lib/python3.8/site-packages/fitparse/base.py", line 156, in _parse_message
    add_dev_field_description(message)
  File "/home/evensolberg/.local/lib/python3.8/site-packages/fitparse/records.py", line 450, in add_dev_field_description
    field_def_num = message.get('field_definition_number').raw_value
AttributeError: 'NoneType' object has no attribute 'raw_value'

[fgebhart]

Thanks for looking into this in more detail. The traceback provided clears things up:

Turns our workoutizer is not able to parse your 2019-04-12-140638-FITNESS 416C-2-12.fit file. Under the hood workoutizer relies on fitparse, which is a great Python library for parsing fit files and which is where the error originates from.

Could be that fitparse does not support fit files generated by wahoo devices. Could also be that wahoo fit files do not adhere to the fit format specifications. But before digging deeper in that direction I would like to rule out that the problem actually originates in workoutizer.

Therefore I would offer to analyze the problem with your fit files on my side a bit further. In case you don't mind sending me one of your files (which caused the above traceback), of course.

[evensolberg]

Sure, I can do that. Just give me a little time. Weekdays are somewhat busy. 😃

[fgebhart]

No worries, take your time 😃

Regarding transferring the file, you could either

• use file.io
• send me an email to [email protected] or
• zip and attach it to a comment here in this thread

[evensolberg]

2019-04-12-140638-FITNESS 416C-2-12.fit.zip

Attached is the first file it's trying to upload. Thanks for your patience.

[fgebhart]

Thanks for sharing your file.

I had a closer look and figured that the issue is actually coming from fitparse. It seems like it is related to this issue.
Which has a fix in version 1.2.0 and workoutizer currently uses 1.1.0 (due to some other migration issues). However, for testing I use both 1.2.0 and the latest master branch and both still failed. This time with a slightly different error, but I assume the root cause resides in this line returning None which leads to a subsequent KeyError in this line.

In order to have your wahoo device supported in workoutizer, it needs to be supported by fitparse first. It would therefore be great if you could open a bug ticket in their tracker.

[fgebhart]

In fact, this issue looks to be the same problem.

[fgebhart]

Closing this for now, since it is not a bug in workoutizer.

Also note, that I added feedback notifications when clicking "Reimport Files" or changing the "Path to Traces Directory" on the settings page (which closes #123) in version 0.18.0.