Your Philly IRS agent is wrong. You need 1040 and 2555EZ. These are available on the internet, or you can buy older versions of TurboTax (and other tax software) on EBay.
You say you're confused about not having to file. Earning under $80,000 but over the various limits ($2900 - 3200) means that you pay no tax, BUT you still have to file. I'm not sure how much simpler the concept can get: You have to file, but no tax is due.
When you owe no tax, you only have to file the past three years, not the past six. The past three years would be 2005, 2004, and 2003.
Good luck!