You can find a great deal of basic information about every model here:
http://battlecollege.wikispaces.com/.
Here is the link to the slightly outdated official pricelist:
http://privateerpress.com/files/product-list-may-2009.pdf.
Current MSRP as well as discount prices can be found here:
http://www.thewarstore.com/PrivateerPress.html.
You can spend as much or as little as you want. In general, you can start off with the battle box for your faction and add one unit and one solo for a complete 500 point force. that will cost you probably just over $100.
However, all you actually need to start is the $50 battle box for your faction. You get one Warcaster, 2-4 Warjacks, and the quick start rules. The special rules for each model/unit are included on the stat cards that come with the models.
For advanced rules, like infantry or cavalry units, you'll want to go ahead and pick up Prime: Remix or Primal (depending on if you choose a Warmachine or Hordes faction). That will cost you an additional $25.
Be aware that first quarter next year Warmachine MKII will be released and will change the core rules. Shortly afterwards, Faction books will come out detailing all of the models thus far, plus some new stuff, for each faction. After the first wave of faction books, the game will revert to its current release schedule of one expansion book that covers all factions.