Don't know that they struggled as much as they eventually overextended...they had some big sellers, Cannonball, Mongo's Watermelon Man...pretty sure that Wes, Griff, Bill Evans sold respectably, maybe even Johnny Lytle.
I don't think the children's label was a tax dodge, though, not if they put quality into it. Or if it was, it was a tax dodge with a conscience.