My husband and I started a second family, and at the time I was 34, nearing 35. I also have a 16 year old son from a previous relationship.
Baby wasn't planned, but he has been an absolute joy. With my first son, he went into day care so that I could work and I missed a huge chunk of his life. With my second son, I am a stay-at-home mum and marvel each day at the things I missed the first go around. I find I am more patient now and better equipped to handle motherhood than I was at 20.
Pregnancy later in life was certainly not as easy as it was in my early 20's. Recovery time took longer and left a few side effects which were not so nice. There was also the thought that I was just about finished with child rearing -- gone were the bottles, diapers and 2am feedings. In less than two years, my first son would be an adult, with the hardest part of motherhood done and over. In a way, I was looking forward to it being just me and my husband -- able to do whatever we wanted to do.
I don't think you are crazy for wanting to start another family. So long as it is something you both want, go for it.