well, but a normal ping to a decent server is betwene 5 and 30 over here, so it all comes down to the overhead for the de/en coding of the stream, transmitting the controls back shouldn't cause much lag if the servers are decently connected.
I think its a very great idea, if the tech works out...