I think you might be asking the wrong question - it's subtle!
Are you looking for quick growth, or largest possible population?
If you're looking for the fastest population growth, then Prestige and an external supply of basic resources will get you going fastest, but research speed is the limiting factor (it cannot be accelerated). If you're self-sustaining, then the lower levels of building offer the best returns on build time. If you're not using Prestige much, then at the beginning an Architect's Office gives good returns on investment up to level 12-ish, then it's best holding back until your buildings are taking 2-3 days per level.
If you're looking for the largest possible population, then the military economic support buildings (Spearmens' Billets, Cavalry Parade Ground, etc) each have 1796 population at level 20. Advanced resource production buildings (Saddlemaker, etc) and basic resource enhancement buildings (kiln, foundry, etc) have 1497 pop. A Traders' Office weighs in at 2000 population. Don't forget that you'll need an adequate food supply to feed that high population, and that taxes will reduce your production so if your city is going to do anything useful (having diplomatic or military capability), you'll need to raise taxes. So, the biggest population does not necessarily make the most useful city.
Edited by Albatross - 19 Mar 2013 at 02:54