I made the switch from NGINX to Caddy. For me, configuring Caddy is much more simple than configuring NGINX. Also Caddy automatically obtains and renews SSL certificates.
So, Caddy's simplicity is what won me over. I don't care about speed since I'm the only user of my self-hosted services.