CALTECH 256, and several others. Many papers presented in CVPR 2012 (http://www.cvpr2012.org/) were indeed on very large datasets.
Do note that some of these datasets are fairly large (may or may not be as broad though).
The algorithms you use will depend on the complexity you can handle. OpenCV, IPP, and CCV (which was on hacker news few days back) could provide good options for algorithms with a training dataset you choose/create.
Data size is indeed one of the key factors to consider when building vision systems, and in this respect, the current champ is Image Net: http://image-net.org/
They have images labeled according to the WordNet ontology, and the dataset is still growing. They also have more detailed annotations on subsets of the data for various things (object, attributes, etc.)
The labelme dataset: http://labelme.csail.mit.edu/instructions.html This is close to what you were thinking (and I liked) -- having annotation done by public.
A list: http://www.computervisiononline.com/datasets
PASCAL VOC: http://pascallin.ecs.soton.ac.uk/challenges/VOC/voc2011/
CALTECH 256, and several others. Many papers presented in CVPR 2012 (http://www.cvpr2012.org/) were indeed on very large datasets.
Do note that some of these datasets are fairly large (may or may not be as broad though).
The algorithms you use will depend on the complexity you can handle. OpenCV, IPP, and CCV (which was on hacker news few days back) could provide good options for algorithms with a training dataset you choose/create.