Hello @akapowl
Apologies for the extremelly late reply, had to put this project on the back burner, but was finally able to fix it.
I ended up using a variation of Jacks approach.
But instead of tweaking the snapping, I added a cheeky 10px on the scrollTo position, which madee the jump disappear.
Here's the fixed pen, for anyone who might have this problem in the future.
https://codepen.io/sueharaluke/pen/poPJypw
Thanks for the help !