I got it. Yes it turns out the pointer is out of sync because I was hit testing the wrapper div element attached to the directive. I was able to fix that by hit testing the inner div the image is attached to instead.
The behavior with the drag item accelerating across the screen only happens in my app (not in the codepen) in iOS/android and desktop Chrome/Safari.. I will have to do some digging there. Thank you for the tip about using the transform object's X,Y.