I usually do that too, when I only have 1 worthless item to trade in. On days when the hermit has 2 clovers, or when I want to get multiple items all at once (especially on day 1), I tend to get the required number of worthless items and then go and buy the required number of permits.
I have the autopermit thing on, but that only buys 1 permit at a time (which is fine), but that then requires more clicks anyway to visit the hermit again.
Edit: Nevermind. You were too quick for me! Thank you very much.