Hey Cassie!
Thanks for helping out with this. I was really trying a total different approach. Before implementing the logic that is there in my question, I had achieved this effect with a TOTAL DIFFERENT APPROACH and now I'm realizing that it was very dumb.
This was very smooth though, but was done using scrollTrigger's markers. This wouldn't work if turn the markers off. (very dumb)
https://codepen.io/abhiishek-10/pen/ZEPQKZL
But anyways, the solution you've provided is really awesome and perfect. Just tweaked a little and achieved what I wanted.
Here's the link to my solution. Hope this helps other people ; )
https://codepen.io/abhiishek-10/pen/abMdWgV
Thanks again Cassie. Appreciate your help.