With that approach, I face difficulties on both desktop and mobile.
On desktop: since it is not possible to attach event listeners to the scrollbars, only to their associated elements, pointermove events (following a pointerdown event) which happen somewhere on the element but outside of its scrollbar would be mistaken for scrollbar scrolling, unless I made some nasty assumptions about scrollbar width, which can vary from browser to browser. Makes for a very brittle solution.
On mobile: a pointerdownevent followed by pointermove events does not necessarily indicate scrolling. If the events land on a child element with touch-action: none, no scrolling would occur.