Anyone can issue a PURGE request for any resource and invalidate your caches. That can lead to increased bandwidth costs but also potential Denial of Service attacks.
Varnish is a reverse HTTP proxy, sometimes referred to as an HTTP accelerator or a web accelerator. A reverse proxy is a proxy server that appears to clients as an ordinary server. Varnish stores (caches) files or fragments of files in memory that are used to reduce the response time and network bandwidth consumption on future, equivalent requests. Varnish is designed for modern hardware, modern operating systems and modern work loads.
Access restriction should be applied.