Well, first thing is, you have to decide what kind of riding you are going to do. Do you want to ride mountain bike trails? Will you ride strictly in the Park? Do you want to do organized rides on the road (aspiring to do the MS150)? Just leisure riding in your neighborhood... or commute to work?
You also might want to consider who you will be riding with and what kind of bikes they are riding. If they're riding road bikes and you purchase a mountain bike or Cruiser, it is unlikely you'll be able to keep up with them and will find yourself defeating your purpose riding by yourself.
There are many different kinds of bikes:
Road bikes are fast and easy to pedal on pavement. They are incapable of operating off the road. Many people find the riding position difficult to maintain for a long time, but with saddle time and practice, comfort comes with proper form..
Mountain bikes are harder to pedal and slower on pavement. But they have a cushy ride, an upright riding position, and can travel easily on a wide variety of surfaces. You can have fun is some crazy place too!
Hybrid or Cross bikes are almost as fast and easy to pedal as a road bike, while being almost as comfortable and versatile as a mountain bike. They still use knobby tires, so there is some wasted energy, but chang'em out for slicks and you'll be set. Also, these bikes will weigh more than road bikes, in general.
Comfort bikes are just that—comfortable. They are less efficient, but sitting on one is much more comfortable.
Cruisers are fun to look at and, when ridden at a relaxed pace, they are ideal for admiring the scenery, exploring the neighborhood or beach.
Recumbents are quite comfortable, very fast. They sit so low to the ground that visibility can be an issue, and they require some learning to be operated with maximum efficiency.
As for me, I started with a mountain bike. I did not have a clue what to begin with, but I knew I did not want to make a huge investment in the unknown. Honestly it looked the most comfortable to me and even though I had no real basis for the choice, I figured it would get me started.
I purchase my first bike, a Bianchi Ibex mountain bike on Ebay. :) I was so proud!!! I found a seller locally who was willing to meet me for the pick-up/drop-off to avoid nasty shipping.
I quickly realized I enjoyed the riding but that for road riding, I had the completely wrong bike. My Mountain bike was heavy, heavy, heavy! But instead of running out and buying a road bike, we (as in my Husband) changed my tires to slicks, and what a difference that made! While it made for an incredible and improved riding experience, I still could not keep up with friends when riding in groups, and I did not see navigating mountain bike trails in my future. LOL
So, not long before my first tri in early 2000, I purchased my first road bike since I was doing the bike transition on a relay team. It was a Specialized Allez, on clearance. I think I paid $399 (originally $599) for it. It was a good entry-level road bike. That bike changed the way I felt about riding, completely, and in a good way. I did plenty of events with this bike and enjoyed cycling more and more with every event.
Sometime before I had my Daughter in 2004, I got a bonus and decided I wanted to upgrade my bike. I paid $1500 for the new one, I think. It's a Specialized Dolce Vita -- a woman's specific bike. Again, this bike made me enjoy cycling even more than the last bike. It fits me well, it performs well and the components are a step or two above my last bike. I think I am happy with this bike and really don't see a more expensive bike in my future. I'm content and if I want to make improvements and change components I can.
Hope this helps!