Well I love Bury. They've got some great shops there, a beautiful cathedral (though my favorite part is the surrounding graveyard - VERY cool). There are some really wonderful cafe's and restaurants as well, and of course the weekly market there. Loads of buildings in Bury date back to Elizabeth the 1st - and it's full of coool historic places to go. THe train station is sort of crappy, but there you go. At least it works.
The National Trust also has several places within driving distance to check out - notably Angelsey Abbey, WIcken Fen (great for hiking or nature-watching), West Stow (a re-created fully functioning anglo-saxon village) - and only about 45 minutes away is Ipswich which has loads of things, and also the Sutton Hoo Burial Ground which is one of the biggest treasure sites in England, the actual treasure is in the British Museum, and you can roam around the huge burial hills and stuff, it's very cool. I've been there a few times.
One of the best things about Bury is it's proximity to other places. Definitely come to Ely - it's a gorgeous Cathedral town that used to be an island back in the God-knows-whens, so when you drive towards it all you see is a great big hill out of nowhere with a cathedral on top.
Ely has 2 weekly markets that are definitely worth going to - the Thursday farmers market and the Saturday arts & crafts market. There's a train station in Ely - easy from Bury - and from Ely it's exactly a one hour train ride into London. You're also within minutes of Mildenhall - which by itself is a truly crappy town that I hate - but you can always go to the pubs and rest assured that you'll run into 12,000 Americans because of the air base there.
And there's always Cambridge. Whee! Great bars, pubs, restaurants, festivals, shopping (a huge Borders book shop among my favorites), and things like punting on the River - and Christmas in Cambridge is wonderful. Again, it's an easy train ride.
Here's my England page, that includes a brief tale of me getting my fiancee visa. The only hitch we ran into (well, I, since David wasn't there to apply with me) was proving that we had a palce to live in England, we ended up having to get his parents to fax the office a copy of their house deed since David & I were temporarily staying with them).
Hope that all helps a bit.