Loading data casues results in cbor decode error

[boreq]

The issue that I will be describing concerns upgrading the library from an older version to the current master version. The version the library is being upgraded from is b4d663e and the new version is the current master which is 7427436.

The new version seems to be unable to load data previously saved by the older version. The following error is logged when starting go-ssb pointed at a repository previously used by the old version:

ts="2022-02-23 18:42:37.2501660 (UTC)" level=warn index=timestamps event="index stopped" err="error decoding data for seq(0) (ofst:0): multiMessage: legacy decoding failed: cbor decode error [pos 3]: invalid number loading uint64, with descriptor: 88 (bytes)"

The exact place where the error occurs is here:
https://github.com/cryptoscope/ssb/blob/74274366774ee86b22cc37c93d689cfd8a700694/message/multimsg/multimsg.go#L115

As far as I can tell this failure occurs when loading data from the ReceiveLog:
https://github.com/cryptoscope/ssb/blob/74274366774ee86b22cc37c93d689cfd8a700694/sbot/indexes.go#L128

My understanding is that the format of the data stored in ReceiveLog has changed over time.

In order to address this I explored several options. I attempted using the existing migration code (namely UpgradeToMultiMessage, which I believe has been removed from the library) however I was not able to successfully migrate the data when using it, in fact looking at the repository version this shouldn't be necessary. I also attempted to modify MultiMessage.UnmarshalBinary by unmarshaling into OldStoredMessage (now also removed) but without success.

I would appreciate any help that could help me overcome this hurdle.

[cryptix]

Hey,

sorry you had to stab in the dark there and the late reply. Was tied to the couch with a fever...

Your assumption is right. I did this change back around #67 - before there were multiple different binary encodings in the messages and before adding another one on top I changed all of them to what is used in private groups/envelope (https://github.com/ssb-ngi-pointer/ssb-bfe-spec, which for history reasons is named TFK in the go code).

At the time no one was working at planetary on the go code and I honestly assumed it would stay that way. I'm glad you are tackling this but because of what I just mentioned I also skipped writing migrations and just deleted my data and re-synced.

The old migration code and tool was for a previous change, so sadly is a bit useless here but might give you a good staring point - basically you need to use what it migrated to as the source and what is now used as the destination format. To get the old struct definitions you need to do some digging and copy 🍝 with the state of this repo before the aforementioned PR, I fear.

cc #122

[boreq]

I am closing this as I don't really know what the resolution of this issue should be apart from introducing some kind of a migration mechanism which is definitely out of scope here.