jillsmetanka, the important thing to realise here if that there is a rule that you must pass .... and UKBA have no discretion about this particular rule ... you either pass it, or your application for Naturalisation will fail!
You are not married to, or in Civil Partnership with, a British Citizen and accordingly for you, you have a 5-year qualifying period. So the rule for you is ... you need to have been physically in the UK exactly 5 years before UKBA receive your Naturalisation application.
You tell us that you got your UPV in 2004, and prior to that you often visited. So, picking a date out just to illustrate, you intend to make your application for Naturalisation on 13.04.09. Question ..... were you physically in the UK on 13.04.04? If the answer is "no" then you cannot make the application, or rather, any such application would definitely fail.
Your dates as a visitor to the UK in 2004, before you got your UPV, do you want to give us details?
Another rule ... in your 5-year qualifying period, will you have been outside the UK for more than 450 days in that 5 years?