If we create file:
templates/current_template/html/mod_virtuemart_category/somefile.php
we will not see it in the list of available layouts in the backend, when edit module parameters
however, if you put the file in the
modules/mod_virtuemart_category/tmpl/somefile.php
it will appear in the select list
This is a Joomla issue. Joomla module overrides (in yourtemplate/html) must use the name of a view file already in the module tmpl directory. Normally there is only one default.php view file. They are not as flexible as component overrides. It may be that all you need to do is add the file.php name into the module's .xml file but I have not tried this.