Actually I think you could make your code more compact if you concat your category id-s, than your query will result only one line or nothing.
So you could do a query like this:
$query = 'SELECT GROUP_CONCAT(`category_child_id` SEPARATOR ", ") FROM `#__virtuemart_category_categories` WHERE `category_parent_id`="'.$vm_catid.'"';
This will return something like this: 1, 2, 3, ...
Oh and your $where[] mod has a syntax error
If you put a comma in the query and you doesn't have a child category than you will get a MySQL error because of that comma.
So you should make your code like this:
$catscont = array();
$this->untreeCat($virtuemart_category_id, $catscont);
$qkat = ' `#__virtuemart_product_categories`.`virtuemart_category_id` IN('.$virtuemart_category_id;
foreach ($catscont as &$kat){
$qkat .= ', '.$kat;
}
$qkat .= ')';
$where[] = $qkat;
Otherwise great work!