Respiration is another way to lose body heat, so if you're not wearing a balaclava, fold a bandana into a triangle and tie it to fit over your nose and mouth—just like the robbers in the old Westerns. This can make the difference between a comfortable ride and a miserable one. Just remember, don't go into a bank like this.
I recommend cycling gloves for a couple of reasons. Most cycling gloves are cushioned on the palms, providing proper circulation in the various hand positions on your handlebars. Gloves also protect your hands from road rash if you fall. In the winter months, full fingered gloves are a good idea or cycling mittens for more extreme conditions.
Since the feet are pedaling circles and churning through the cold air more than the rest of your body, they need to be protected from the cold. Like the head, body heat is lost to a large degree through the feet. For cold-weather riding, use a heavier thermal cycling sock that wicks moisture and retains heat; choose socks made from synthetic fabrics.
Cycling booties slipped over your shoes are great in cold weather. The booties are designed to accommodate your pedal cleats, and insulate your foot and ankle as well. For days that aren't cold enough for booties, wear toe covers. Toe covers accomplish the same thing that a windbreaker does for your chest: they keep the cold air from penetrating your foot.
If you're on a ride without booties or covers and your feet get cold, get plastic bags from a convenience or grocery store and slip them over your feet (inside your shoes). While you're at it, you might as well ask for a hot cup of coffee—the hot coffee will help heat up your core from the inside.
Another rule of thumb is to keep knees covered anytime the weather is below 50 degrees. This helps keep them warm and protected from the cold air, thus keeping them properly lubricated and functioning. For semi-cold weather, short cycling tights are a good option; they come down just below the knee without covering the entire leg.
Cycling leg warmers are also very convenient as they are easy to zip on and off quickly as needed. For colder weather, full cycling tights range from lightweight to heavy and waterproof, or you can find insulated cycling pants.
Like cycling tights and pants, arm warmers can be used to keep arms warm in semi-cold conditions. Arm warmers can be quite useful for days that start out cold and warm up as the sun comes out. As the weather warms, the arm warmers can be rolled down or taken off and stored in your jersey pocket. Again, there are varying degrees of thickness and insulation of arm warmers.
On winter days that don't warm up, wear a long-sleeved jersey (either insulated or not depending on the cold) for your middle garment as described above.
Now that we've gone over the cold-weather apparel to protect the body, don't forget the basics: A proper helmet, good cycling shorts with chamois and proper cycling shoes.
Now, adequately armed with what it takes to stay warm in the cold, you're ready to hit the road year round. You'll love it and your fair-weather cycling buddies will be envious. So just get out there and ride. Don't let a cold day keep you inside, just prepare for the cold properly.