Let's Encrypt privacy concerns for private networks

I vaguely remember reading something about leaking your private network setup if you used Let's Encrypt to generate your certificates.
Because of this when I installed my reverse proxy with caddy to handle my selfhosted home network I configured it to generate the certificates locally.
But this comes with the issue of the annoying warnings of the browsers plus being unable to connect to those devices/services which can't ignore it.

Am I being too paranoid? Is there any real concern about generating the certificates with Let's Encrypt for addresses which I don't intend to have outside my private network?