I'm 5'4" (ie. not tall), bike 5 miles a day and do all of my communting on foot or by bike. I eat 3 meals a day, I don't drink soda, I don't "snack", I don't drink beer... I've done nothing to "let myself go", this is who I am.
I'm afraid a great deal of what determines your body shape is genetic and related directly to inherited factors that you can not control (metabolism, thyroid issues, your parents' body shape, their parents' body shape, etc). Thanks to my SKELETON, I would never fit into any jeans under a size 6 reguardless of how much weight I lost... because I have a pelvis. To imply that everyone over a size 4 is "fat" and letting themselves go says to me you've bought into the proliferation of the image that everyone needs to be tiny to be attractive.