Yah, I would avoid stuffing the cavity. If you want to stuff the turkey then stuff the neck.
You can either, get a wooden spoon and carefully part the skin from the turkey breast to create a pocket over the turkey breast, then put the stuffing up there, fold the neck skin up to keep it in and then secure it with a tooth pick, or you just stuff the neck cavity without pushing it up over the turkey breast and then secure it again by folding up the spare skin and securing it with a tooth pick.
The other thing you could do is, part the skin from the turkey breast with a wooden spoon and then pack herbed butter in there, it's a good way of keeping it moist and making sure all the butter melts into the breast, I also sometimes but streaky bacon up there, it's much nicer than putting it on the outside as the bacon wont dry out, and you get crispy skin still.