Based on suggestion from Aron Budea.
And do something similar to most other places keeping vectors
of weak references where the code looks like it will hold more than
a few entries.

Measurements:
the 26 page file file takes
    51s without my path
    15s with this patch
the 69 page file file takes
    5m28 without my path
    51s with this patch
the 84 page file file takes
    8m28 without my path
    58s with this patch

Change-Id: I8da94c525fc73ebd969e0343c6f074be4f0063b1
Reviewed-on: https://gerrit.libreoffice.org/27093Tested-by: Jenkins <ci@libreoffice.org>
Reviewed-by: Noel Grandin <noelgrandin@gmail.com>