For the speed at which the particles rise: you have velocity settings for them which you can see in your screenshot. If you turn them down, they'll move slower.
Another possibility would be to have a value > 0 for Damp under Physics > Forces in the Particle Settings.
For the interaction with the chimney: I guess the chimney is already a Collision object, so to take the size of the smoke into account, you have to enable Size Deflect under Physics > Deflection. By default this is turned off and only the center of a particle is used for the collisions.
However, with randomly varying sizes you can get the problem that a particle is born which is too large to "escape" the chimney. I personally wouldn't mind too much if a larger particle goes "through" the rooftop of the chimney as smoke could also wrap around obstacles. Of course this wouldn't look perfect, but since you only have spherical particles and no real smoke, they won't deform around collision objects.
Physics > Forces and Deflection: