I would start here
http://www.antenocitisworkshop.com/wargaming-materials-tools.html If you can't find it there, check something like Secretweapon Miniatures (they have a scenics range). If you still can't find what you are after just go through eBay, there is a bunch of stuff you can find there.
For a more detailed explanation of what you need, the truth is it really depends on what you are looking for. My first suggestion is if you are looking to do more complex scenic bases or just looking to start out, I would suggest investing in a good modelling putty, something like Milliput works, or once you get a bit faster at things I would look at using Kneaditite since it cures much faster. The putty will allow you to fill gaps and push rocks and other pieces together in a more natural way.
For the standard basing material;
Sand - I normally use river sand that I bake in the oven at about 250 C for about a half an hour to dry it out and kill anything. If you really want to buy something online, you could look at gale force 9 basing sand, or maybe model railway ballast.
Rocks - Fish tank pebbles can be good, you can also use broken/ground up cork or sprue (turns out pretty nice for rocks). You could check larger railway ballast too.
Larger rocks - Slate pieces can be found for relatively cheap and carefully broken with a hammer, you can also use large pieces of cork and wood chips.
Grass and flowers - Minatur. Their grass tufts and flowers come in a huge range of colours, you get a decent amount and are absolutely amazing. Army painter flowers and ivy are also pretty good, although you do get less. I use both ranges fairly frequently and would definitely recommend them.
Leaves - Birch Tree Seed pods make for great fallen leaves, and depending on the time of year you can get them in all colours
To do it yourself, you need to invest in plasticard:
Heavy Card Stock - Can be used to build walls, posters, or just generally strengthen things
Cork Sheets - Both 3 mm and 6 mm are useful for various things, even if it is just boosting up height.
Plain plasticard sheets - These can become absolutely anything that you can think of, bricks, tiles, flooring. You can buy textured ones too, but I find it hard to match scale and you can normally carve them yourself. The only sheets I would suggest getting are diamond/checkerplate as its hard to find.
Plasticard Rods and Tubes - Pretty straight forward, you can make rivets using solid rods, and the tubes can be piping or anything else you can dream of.
[list]Other plasticard - I would also suggest picking up some I and H beam shapes too
Balsa wood - Is also a great option, can simulate wood planks quite easily, and be used in structure building
Jewellers Chain/wire - Replicates chains and wires funnily enough
Guitar strings - Used to recreate tubed wiring, although there are now easy to use sculpting tools available to do this with green stuff
Finally we get to trees. These can get pretty complex as to get truly good ones you need to go scavenging a bit. Your first option is to buy model railway trees in various scales (but I find they look obviously fake). Secondly, you can make your own, there are numerous tutorials from using sponge, to using wire armatures and certain types of flocking (both of which can look good, but still very tabletoppy).
To get the very best/realistic trees, most scale modellers and table top guys actually use weeds or grass roots that they pull from the ground, hang to dry and then seal with a mix of PVA and paint. The fibrous nature replicates bark quite well at the scale we are talking about. You can also use smaller twigs/sticks too.
Hope that gives you a bunch of ideas.