I wouldn't call that physical reality dictating your actions, I would call that physical reality limiting options that you are free to choose. For this topic we should narrow it down to very simple and basic actions like raising a hand, or deliberately blinking or holding you breath. Buying a house, choosing a major or things like that have many factors that need to be considered. It complicates matters unnecessarily.
If you choose to raise you left arm instead of your right arm, all things being equal, is it an act of free will or is not? Or, if I ask that you close your eyes and count slowly to 3 - and assuming that you have done that - would you say that you acted out of free will or would you not? If not, then why?
@Karim
If you deny the existence of free will, then I can understand why you would classify your position as being on the unpopular side. However you say this:
That depends on what is included in 'all'. If 'all' includes my choice of doing something, then I can't see how it isn't self-contradictory, and more than just seemingly paradoxical.
I can accept that I can't choose my desires. I can even accept that I can't change my desires (through no ordinary means or perhaps not at all).
I can't accept that I can't choose whether or not to satisfy my desires, all things being equal. Right now I desire to see a movie. I choose not to. I still desire it though.