Given the situation described above, and with little or no tax due, the answer is almost certainly no, you shouldn't worry about your passport. Enjoy your holiday, but get your return filed ASAP otherwise.
You may have heard about the FAST ACT, signed into law by Pres. Obama in Dec. 2015. It does allow for a passport to be withdrawn for non-payment of tax, but the bar for tax due is high ($50,000) and there are a number of actions which must take place first.
Not normally a problem for those resident in the US, but it is something that, theoretically, could hinder those resident abroad. According to a GAO report of last year, IRS mail to a foreign address stands about a 30% to 40% chance of being incorrectly addressed and not being delivered. With $10,000 penalties on failure to report foreign accounts, the $50,000 bar could be quickly attained.
There are no reports of any such action yet, but imagine living in the UK and being in Japan when the immigration officer in Tokyo informs you your US (and only) passport is no good, and the US Embassy in Japan will only issue a temporary passport for travel to the US.
Again, enjoy your holiday and don't worry.