Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Report client status #45

Closed
tobiasKaminsky opened this issue Sep 1, 2023 · 14 comments
Closed

Report client status #45

tobiasKaminsky opened this issue Sep 1, 2023 · 14 comments
Labels
enhancement New feature or request ❄️ 2023-Winter

Comments

@tobiasKaminsky
Copy link
Member

tobiasKaminsky commented Sep 1, 2023

What we agreed on:

  • clients check for capability
  • clients report only if a problem exists, if not, nothing is sent. This also means that if there was an error and error is gone server will not be informed, but can figure it out by last sending date (older than x days no response, but token active -> everything fine again)
  • Android (Report client health android#12009) collects
    • any sync conflicts (with timestamp when it occured)
    • upload with error type:
      • CONFLICT_ERROR
      • UploadResult.MAINTENANCE_MODE
      • UploadResult.CREDENTIAL_ERROR
      • UploadResult.CANNOT_CREATE_FILE
      • UploadResult.SYNC_CONFLICT
      • UploadResult.UNKNOWN
      • UploadResult.FOLDER_ERROR
      • UploadResult.NETWORK_CONNECTION
      • UploadResult.SERVICE_INTERRUPTED
      • UploadResult.VIRUS_DETECTED
  • iOS: Report client status ios#2634
  • Desktop: Report client status desktop#6124

Brainstorm together how/what/when we can send status/health information from clients to server.
Scenario: company with IT department, that wants to have an overview over faulty clients

Already available to server:

  • how many (and which) devices per user
  • first "registration"
  • last usage

What client could do

  • send at fixed time (e.g. once a day) status report
  • send only on error (e.g. sync error on Desktop)

Server: nextcloud/server#40160

@AndyScherzinger @jancborchardt

@jancborchardt
Copy link
Member

We should also loop in @sorbaugh and/or a relevant engineer here. :)

@tobiasKaminsky
Copy link
Member Author

collect local problems: writing problems, disk full, critical upload/download errors?
- send this with timestamp

@tobiasKaminsky
Copy link
Member Author

tobiasKaminsky commented Oct 4, 2023

Android collects

  • any sync conflicts (with timestamp when it occured)
  • upload with error type:
    • CONFLICT_ERROR
    • UploadResult.MAINTENANCE_MODE
    • UploadResult.CREDENTIAL_ERROR
    • UploadResult.CANNOT_CREATE_FILE
    • UploadResult.SYNC_CONFLICT
    • UploadResult.UNKNOWN
    • UploadResult.FOLDER_ERROR
    • UploadResult.NETWORK_CONNECTION
    • UploadResult.SERVICE_INTERRUPTED
    • UploadResult.VIRUS_DETECTED
  • E2E errors

@camilasan @allexzander @claucambra @mgallien for Desktop
@marinofaggiana @mpivchev for iOS

Can you also add your ideas what you want/can send in case of problems to server?

@alperozturk96
Copy link

Server creates table under user table such as user_upload_statistics, user_app_events and expects statistic data (e.g UploadResult, Events).

Client sends UploadResult to server for related remote calls.

Maybe even more when user navigate the media screen we can also send events to server (e.g Events.mediaScreenOpen)

Therefore we can track faulty clients and optionally have a good overview about general app usage. This can be useful for some of our clients. @tobiasKaminsky @AndyScherzinger

@tobiasKaminsky
Copy link
Member Author

For privacy reasons we do not want to track/send successful operations, but only faulty states.

@marinofaggiana
Copy link
Member

  • E2EE:
  • errorE2EENotEnabled
  • errorE2EEVersion
  • errorE2EEKeyChecksums
  • errorE2EEKeyEncodeMetadata
  • errorE2EEKeyDecodeMetadata
  • errorE2EEKeyVerifySignature
  • errorE2EEJSon
  • errorE2EELock
  • errorE2EEEncryptFile
  • errorE2EEEncryptPayloadFile
  • errorE2EECounter
  • errorE2EEGenerateKey
  • errorE2EEEncodedKey
  • errorE2EENoUserFound
  • errorE2EEKeyCiphertext
  • errorE2EEKeyFiledropCiphertext
  • FORBIDDEN
  • ROSOURCE_NOT_FOUND
  • CONNECTION_LOST
  • BAD_SERVER_RESPONSE
  • CHARACTERS_FORBIDDEN

@marinofaggiana
Copy link
Member

Android collects

  • any sync conflicts (with timestamp when it occured)

  • upload with error type:

    • CONFLICT_ERROR
    • UploadResult.MAINTENANCE_MODE
    • UploadResult.CREDENTIAL_ERROR
    • UploadResult.CANNOT_CREATE_FILE
    • UploadResult.SYNC_CONFLICT
    • UploadResult.UNKNOWN
    • UploadResult.FOLDER_ERROR
    • UploadResult.NETWORK_CONNECTION
    • UploadResult.SERVICE_INTERRUPTED
    • UploadResult.VIRUS_DETECTED

@camilasan @allexzander @claucambra @mgallien for Desktop @marinofaggiana @mpivchev for iOS

Can you also add your ideas what you want/can send in case of problems to server?

@tobiasKaminsky please write the code error near you internal code string (every o.s. use a different system but the same error code)

@marinofaggiana
Copy link
Member

marinofaggiana commented Oct 13, 2023

Is not clear:

Send the error for single upload where counter > 1 ? (some errors occur for individual uploads others for each upload, so more information should be sent) or if need only an alarm every upload failure >= 1 time should be transmitted, and one successful upload is enough to cancel everything and transmit nothing (no loop, example disk full).

@tobiasKaminsky
Copy link
Member Author

Hm. For now I would say that we send each error category on its own, o.eg.:
CONFLICT_ERROR: 2, timestamp
VIRUS_DETECTED: 1, timestamp

They then can be combined on server, if needed.

@marinofaggiana
Copy link
Member

can you write the errors number so all use the same ?

@marinofaggiana
Copy link
Member

nextcloud/ios#2637

@allexzander
Copy link

allexzander commented Oct 25, 2023

I will come up with my ideas for desktop client later. For now, I think we must come up with a standardized way of collecting errors on the server, like have an SQL table with columns we must fill for every client app and also some kind of metadata column for platform-specific info. My idea is we need to have a list of error types(it can be long to include most common scenarios and we can add more error types later. Tobias already mentioned this. I would go even further and also include file full path or whatever unique id the file have such that file full path can later be obtained on the server. That way, we can later have some kind of view in the WebUI where we could sort errors by different columns (error type, timestamp with selection for specific time range, filename, folder type (group/normal/external storage), and also have repeating error type for the same file collected such that it is not displayed 1000 times (1000 rows for the same file for the same error) but instead is displayed as one row with a number of occurrences with the ability to open a detailed view with all 1000 rows when needed).

For example, we have server logs that can be viewed in Web UI in that fashion. This is something I would target for. Then those errors can be exported in the form of a text file or CSV file or something.

UPDATE: As I mentioned in my ideas in corresponding desktop client issue, we may also reduce spam and stop sending the same error for the same file in a given timeframe more then once by keeping track of these errors in the desktop client's local database, but in this case we might lose the ability to have detailed reports on the server.

Also, as mentioned in the desktop client's issue ideas, I would consider sending a client log (only the current log file) to the server in case of sync errors, as client logs are being rotated constantly and important log might get lost which will prevent debugging.

@tobiasKaminsky
Copy link
Member Author

@allexzander can you add your categories/problem names here?

@allexzander
Copy link

@tobiasKaminsky

DownloadError

  • DownloadError.CONFLICT
  • DownloadError.CONFLICT_INVALID_CHARACTERS
  • DownloadError.CONFLICT_CASECLASH
  • DownloadError.CANNOT_CREATE_FILE
  • DownloadError.SERVER_ERROR (with code and message in metadata)
  • DownloadError.VIRTUAL_FILE_HYDRATION_FAILURE (with full description/reason in metadata)
  • DownloadError.NO_FREE_SPACE

UploadError

  • UploadError.SERVER_ERROR (with code and message in metadata)
  • UploadError.NO_WRITE_PERMISSIONS
  • UploadError.NO_FREE_SPACE

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request ❄️ 2023-Winter
Projects
Archived in project
Development

No branches or pull requests

6 participants