I don't know why the 'Slow Parent' is causing a change in scale - intuitively that just doesn't make sense. Perhaps it's worth logging it as a bug report.
However, you can achieve the same effect using Drivers, using a Scripted Expression to define the relationship between the two objects.
To set this up, right-click the Rotation field in the properties panel (N) and 'Add Driver'. This should make the field turn purple to indicate the presence of the driver.
Swap to a Graph Editor window and change the mode to 'Drivers'.
Select your driver and you should see a variable ('var'). Add a second variable and set each variable to the relevant property in each object. In my example I have each set to 'rotation_euler1' for each of Cube and Empty.
Set the scripted expression to '(var * 6 + var_001)/7 if frame>1 else var_001'. This expression uses a conditional ('if .... else') to set the property to the value of 'var_001' on frame 1 or to the weighted average of 'var' and 'var_001' for any subsequent frame.

NOTE : You may need to set the 'Auto Run Python Scripts' in the User Preferences to enable 'Scripted Expressions' in Drivers if not already set.

This produces the following result :

To get the child to follow the parent more or less closely you can adjust the values in the weighted average. For example, '(var + var_001)/2 is an 'unweighted average', '(var * 2 + var_001)/3' is more weighted towards 'var', (var * 3 + var_001)/4' is even more weighted towards 'var'. To generalise, set it to '(var * A + var_001 * B)/(A+B)' and adjust A and B as desired.
Blend file : 