that believability (sic) may well suit a casual observer. But the number of clients, and the minimising of tax liability by using dividends, have no legal bearing whatsoever on IR35.
Not until a judge agrees with your believability at a tax tribunal.
I'm sure that if you pay yourself £40k PAYE and small dividends you are a lot less likely to get investigated, but if I was that worried I'd buy QDOS TLC35 insurance as it's cheaper.