Skip to content
This repository has been archived by the owner on Mar 25, 2024. It is now read-only.

export fetched data into a csv file #777

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

export fetched data into a csv file #777

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5525938
export fetched data into a csv file
Vebryn Aug 23, 2017
a867c06
clean old exportToCSV method
Vebryn Sep 17, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions app/src/main/java/nodomain/freeyourgadget/gadgetbridge/devices/AbstractSampleProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,14 @@
import java.util.Collections;
import java.util.List;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import de.greenrobot.dao.AbstractDao;
import de.greenrobot.dao.Property;
import de.greenrobot.dao.query.QueryBuilder;
import de.greenrobot.dao.query.WhereCondition;

import nodomain.freeyourgadget.gadgetbridge.database.DBHelper;
import nodomain.freeyourgadget.gadgetbridge.entities.AbstractActivitySample;
import nodomain.freeyourgadget.gadgetbridge.entities.DaoSession;
Expand Down
1 change: 0 additions & 1 deletion app/src/main/java/nodomain/freeyourgadget/gadgetbridge/devices/SampleProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,6 @@
import android.support.annotation.Nullable;

import java.util.List;

import nodomain.freeyourgadget.gadgetbridge.entities.AbstractActivitySample;

/**
Expand Down
11 changes: 11 additions & 0 deletions ...reeyourgadget/gadgetbridge/service/devices/miband2/operations/FetchActivityOperation.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.File;
import java.io.IOException;
import java.text.DateFormat;
import java.util.ArrayList;
Expand Down Expand Up @@ -57,6 +58,8 @@
import nodomain.freeyourgadget.gadgetbridge.service.devices.miband2.AbstractMiBand2Operation;
import nodomain.freeyourgadget.gadgetbridge.util.ArrayUtils;
import nodomain.freeyourgadget.gadgetbridge.util.DateTimeUtils;
import nodomain.freeyourgadget.gadgetbridge.util.FileUtils;
import nodomain.freeyourgadget.gadgetbridge.util.CSVExport;
import nodomain.freeyourgadget.gadgetbridge.util.GB;

/**
Expand Down Expand Up @@ -207,6 +210,14 @@ private GregorianCalendar saveSamples() {

timestamp.add(Calendar.MINUTE, 1);
}

// export csv on fetching
File myPath = null;
myPath = FileUtils.getExternalFilesDir();
File myFile = new File(myPath, startTimestamp.getTimeInMillis() / 1000 + ".csv");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe add a gadgetbridgedb_ prefix (or something like that, maybe shorter)?

But really, this does not belong here. It should be implemented in one place for all devices, not in the Mi Band 2 FetchActivityOperation. And I guess we should also have an option to turn this on and off.

Copy link
Contributor Author

@Vebryn Vebryn Sep 17, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, does data from other devices differs ? Where would you place on/off feature ? I would probably add into Mi Band menu since it's the only device that's supporting it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Data from other devices may differ, yes. So either we use normalized values (= the same data ranges for all devices) or device specific raw values, or even both.

But it would be good to have this for all devices, not just one.

Since some devices provide the data in a different way, where we have no access to all samples at once, we might call a method like samplesAdded(device, start, end) which would then check if CSV export is enabled, then fetch the samples from the given date range from the db (probably cached in the session already) and perform the export.

Alternatively we could integrate with the new DB export functionality and perform the CSV export repeatedly. We would just need to remember the start and end dates so that we only export the incremental changes.

CSVExport.exportToCSV(samples.toArray(new MiBandActivitySample[0]), myFile);

// store samples on fetching
sampleProvider.addGBActivitySamples(samples.toArray(new MiBandActivitySample[0]));

saveLastSyncTimestamp(timestamp);
Expand Down
42 changes: 42 additions & 0 deletions app/src/main/java/nodomain/freeyourgadget/gadgetbridge/util/CSVExport.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
package nodomain.freeyourgadget.gadgetbridge.util;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;

import nodomain.freeyourgadget.gadgetbridge.devices.SampleProvider;
import nodomain.freeyourgadget.gadgetbridge.entities.AbstractActivitySample;

/**
* Created by Vebryn on 29/05/17.
*/

public class CSVExport {
private static final Logger LOG = LoggerFactory.getLogger(SampleProvider.class);

public static void exportToCSV(AbstractActivitySample[] activitySamples, File outFile) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's sufficient to use ActivitySample here instead of AbstractActivitySample.

String separator = ",";

LOG.info("Exporting samples into csv file: " + outFile.getName());
try (BufferedWriter bw = new BufferedWriter(new FileWriter(outFile))) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FileWriter always uses the local platform encoding, which, atm, is UTF-8 on Android. Relying on the local platform encoding will in many cases lead to a bug at some point. One has to use the extra verbose
new OutputStreamWriter(new FileOutputStream(outFile), StandardCharsets.UTF_8))

I've spent too much of my lifetime on encoding bugs already (I actually have a T-Shirt with the text "Schei? Encoding".

For the record, Excel' CSV import does CP-1252 (similar to ISO-8859-1), LibreOffice can handle all kinds of encodings including UTF-8.

bw.write("TIMESTAMP" + separator + "DEVICE_ID" + separator + "USER_ID" + separator + "RAW_INTENSITY" + separator + "STEPS" + separator + "RAW_KIND" + separator + "HEART_RATE");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't we also export the normalized values?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Normalized values are deduced from a simple matching rule. I think correlation have to be done after exportation.

bw.newLine();

for (AbstractActivitySample sample : activitySamples){
String line = sample.getTimestamp() + separator + sample.getDeviceId() + separator + sample.getUserId() + separator + sample.getRawIntensity() + separator + sample.getSteps() + separator + sample.getRawKind() + separator + sample.getHeartRate();

//LOG.debug("Adding line into buffer: " + line);
bw.write(line);
bw.newLine();
}

bw.flush();
} catch (IOException e) {
LOG.error("Error related to " + outFile.getName() + " file: " + e.getMessage(), e);
}
}
}