Need help doing ollie 180 while moving.?

When I try to do a 180 while moving at a decent speed, it feels like me and my board have to go a meter backwards to do a FS 180 if you know what I mean. Its like I'm just doing a pivot that's making me go a meter backwards and it feel awkward doing them down a ledge. Anything to help me out with this ?