A major factor is probably that in fiction authors want memories to work narratively so they make the choice that memories are coherent. The writing would probably turn out a lot more "experimental" if you tried to convey average memories, like where you "remember" for example a place in vague generality, and maybe one or two details but it is all rather disconnected, and maybe doesn't have any particular narrative "scene" to go with it at all. That wouldn't work well in a story, because you have neither a narrative nor evocative details.