Recovering Cursed .bsp files

[snake-biscuits]

When trying to clean up MegaTest results for DDayNormandy's mappack a number of Issues appeared

The most common: all null bytes were replaced with spaces, this renders the map unplayable
b"&\0\0\0" v38 IBSP became b"& " v0x20202026 IBSP, this doesn't seem reversible
replacing all b"\x20" bytes with b"\x00" could do just as much damage as the initial mistake
however, the .bsp files are theoretically possible to "autopsy" by correcting the header then reading in bsp_tool

So long as version control is used and the source .map/.vmf files are preserved this sort of error shouldn't be a huge issue
Still, bsp_tool already gives a window into how a given .bsp might be "broken"
Building on this element, should file recovery be a development focus?

Decompiling is certainly not a focus, though it has helped with MRVN-radiant & understanding titanfall formats

At this present, it might be a good idea to either remove the DDayNormandy mappack from tests.
Or send a pull request cleaning up the pack. Though it's unlikely anyone else uses the mappack, I think?

Other issues with the mappack exposed a difference between shared.Entities & game-engines
It turns out game engines aren't terribly strict with the format
This means all sorts of hacked-together .ent files might exist in the wild
And we'll likely need to refactor shared.Entities to be less about edge cases & more general in it's behavior

Relaxing the dedication to non-destructive editing (preserving the quirks of un-edited data) would make this easier
Though this approach does guarantee (I hope) loading & saving a .bsp w/ bsp_tool does not affect it's play-ability
And automated testing of play-ability sounds like an absolute nightmare, so I'd like to avoid that.

[snake-biscuits]

TL;DR not sure if I should drop the edge-case filled DDayNormandy mappack or refactor shared.Entities

shared.Entities has a lot of clunky edge-case dodging code in it
refactoring while maintaining support for weird maps is a real pain

for the most part I want to ensure that at least any map or lump that would work in-engine loads in bsp_tool without errors
giving warnings for abnormal data & methods to move that data towards standard might help
however, that is also a lot of work.

This is a question about how bsp_tool functions, has functioned in the past & will function in the future.
I don't expect to find an easy answer to this question, or get an answer soon.
Everything here depends on how users want bsp_tool to function, which will change over time.