I know it is not optimal, I was not who built the site, but it is there, it was working before VM 3.8
BTW I had a quite similar issue with a non-Joomla site, there was a kind of parent-child tree solution assigned to the items which is quite similar how VM handles categories. When I worked with this, it took more than a minute to build the parent-child tree with a recursive function and a query in every loop. If I'm right this is how VM handles this. I refactored the code, fetched the tree elements with a single sql query and built the tree with a recursive function but without internal queries, there was only array/object handling with php scripts. I got the same result as above but in less than 2 seconds.
So maybe it does worth some effort do build more scalable code
