Try to keep the memory usage low more aggressively

Memory usage is minimize()d only when a big allocation is done. This could
be problematic for applications that only perform small objects
allocation, as their memory usage could grow a lot, specially if the eager
allocation option is used.

Trigger memory minimization even for small objects allocation to avoid
that pathological case.