I think this related to some bug in setting the initial scroll position. Still don't know why, but when I refreshing it, it works...
So I put this in the useEffect :
setTimeout(() => {
ScrollTrigger.refresh();
console.log("scrolltrigger refresh!");
}, 1000);
It refresh the scroll trigger after the delay, and reset the all scroll position as I want.
It fixed the issue for now...
Adrien Snoos's post in ScrollTrigger on React - Have to refresh to make it works was marked as the answer