saves about 0.5s out of a 43s load

And remove the comment in EndListeningAll,
SfxBroadcaster::RemoveListener doesn't have any weird side-effects any
more

Change-Id: Id7c8ac1bed8ff3487cb8f977990d8fac351d7f03
Reviewed-on: https://gerrit.libreoffice.org/72396
Tested-by: Jenkins
Reviewed-by: Noel Grandin <noel.grandin@collabora.co.uk>