Institutional investors, which have traditionally made up private debt’s largest pool of money, are no longer a source of growth for the $1.7 trillion industry.