You bring up some good points, but Xavier is in Cincinnati, so not really "Dayton" that is supporting them. But when you consider the Cincinnati-Dayton area has the three schools you mention, plus UC (without clicking on the link, but I'm guessing is also over 10,000) shows that Southwest Ohio is more crazy about college basketball than Northeast Ohio.
Part, of that, however, is due to the region not having an NBA team, and in Cleveland's (TV market) case an NBA team that has been among the top 5 most popular in the league ever since LeBron was drafted. Even in the four years LeBron left, the Cavs still were in the top half in attendance and had some of the highest local ratings in the NBA. In the LeBron years, the Cavs have always been in the top five in attendance and have battled the Spurs, Thunder and now Warriors for the No. 1 spot in local TV ratings. I'll have to look up what the Cavs' final local ratings were this year, but saw a couple months ago, they were pulling in something like an 11 share locally, which is about 110,000 TV sets a night ... on top of the 20,562 that go to every home game (and they play 41 of those).
There's only so much time people in NEO want to dedicate to basketball, and largely the area has chosen to invest it with the Cavs, and can you blame them? Consistently one of the best teams in the league with the best player in the world being from right here in Akron. It just is what it is.
With that said, I think there is still potential if Groce builds a product that wins and wins in an exciting fashion where they could bump up to 4,000-5,000 consistently. But as long as Akron is in the MAC and the Cavs are winning, the school simply will have an attendance ceiling it will bump into.
EDIT: See B&G already pointed out Xavier.