But even using NVDEC I'm not happy with performance. It's not bad at all, but not smartphone-like smooth. What can I do to dramatically improve it?
The trick you use make sense, at least is simple. Each time you do a flushing seek, there is a lost state, so you end up ensuring that last seek is completed before issuing next one. You could do the same by using bus messages, that would allow for better compression of the seek location.
My usual approach is to have a state, so when user grabs the seek handle, I enable scrubbing mode. In this mode I issue a seek each time the previous seek completes, I usually don't select accurate seek for speed. Then when the seek handle is released, I issue a flushing seek immediately with accurate flag this time.
That being said, GStreamer often lack a bit of optimization in term of seek performance, contribution are welcome.