Avoid copying arrays

[dannywillems]

          Yes we could slightly optimize the code, but it comes at the cost that we have to always deal with the index translation by 1. This has led to bugs in the previous code base, so I'd like to keep it simple for now and always to the conversion to/from an ml array as a separate step.

In theory we could use array.unshift(0) at the end to get the same result without copying the array. The only reason I don't do that is because TS doesn't understand it

Originally posted by @mitschabaude in #140 (comment)