Fix PartialPriorFactor (again) #3017

Job	Run time
ubuntu-clang-deprecated Debug	22m 32s
ubuntu-clang-deprecated Release	22m 53s
ubuntu-clang-quaternions Debug	22m 25s
ubuntu-clang-quaternions Release	23m 37s
ubuntu-clang-tbb Debug	24m 54s
ubuntu-clang-tbb Release	26m 34s
ubuntu-clang-cayleymap Debug	21m 43s
ubuntu-clang-cayleymap Release	23m 45s
ubuntu-clang-system-libs Debug	22m 6s
ubuntu-clang-system-libs Release	23m 23s
ubuntu-no-boost Debug	21m 47s
ubuntu-no-boost Release	23m 17s
ubuntu-no-unstable Debug	17m 42s
ubuntu-no-unstable Release	18m 49s
	5h 15m 27s

Provide feedback