Figured it out. The blocks/sections above the pinned section had lazyloaded images, which when loaded, changed the document height. That changed the top position of the pinned section which resulted in a wrong sticky/pinned starting position.
We made sure the height of the images are always present, so that the document height doesn't change.