It's less of an issue on the guy 5 Millers deep and more an issue of the other players at the table possibly miscounting a player's stack size for the purpose of effective play. I don't care if you think you've got a barrel of $5s but are actually $12 short of that, but I need to know that when determining bets, considering SPR, etc. Or vise versa, I attempt to put you all in thinking you have less than you do. This is the issue with dirty stacks.
e: And yeah, when you know it's something being shown to you, it stands out and can be identified quickly. But players aren't looking that closely at every chip in a player's stack when playing, they're glancing around and see "okay he's got two barrels of $1s, a barrel of $5s, a couple $20s" and acts accordingly. If the dirty stack doesn't jump out IMMEDIATELY at a quick glance it can be an issue, and the argument is that these, while identifiable, are not going to be easily spotted at a glance when mistakes are made.