This was also my experience. Years 1 and 2 covered general programming concepts like data structures and algorithms. Years 3 and 4 transitioned into lower level topics like compiler design, some embedded work, lots of math. However, as most CS depts at the time we're relatively young, they had gaps to fill, which usually meant adding on more math courses to the curriculum to fill credit requirements.
I am sure these days CS depts are far more comprehensive than 15, 20 years ago and can offer many more courses in advanced programming topics (for example big data, enterprise, security, distributed, project management, mobile, etc).
interesting! I had not really given much thought on how distributed computing or enterprise architectures would fit in a modern curriculum...
I don't think that math should be considered filler though - it was very fundamental to my degree! In fact the first year of my comp. sci. degree was /exactly/ the same as the first year engineering curriculum. It was almost entirely math, with basic programming (just pascal) which the engineers also needed for their degree.
As such I'd say that math is still very prominent in most comp sci degrees
edit: in fact as chance has it I was just today going over some old lecture notes from my 3rd degree year, and even the programming courses were extremely heavy in inductive reasoning to demonstrate how, for example, Big(O) notation worked with different algorithms (i.e., actual formal proofs)
I am sure these days CS depts are far more comprehensive than 15, 20 years ago and can offer many more courses in advanced programming topics (for example big data, enterprise, security, distributed, project management, mobile, etc).