Per issue 593, append .yml to each filename used in the generated configmap. I wasn't able to see alerts in the Prometheus UI until I did this