At $20 in California, I don't think he is a price gouger...just the crazy tax on top of MSRP. I try and do the same thing as you do, when at a local BM with a lounge, I buy their smokes if I am hanging out.
I pretty much concur here. I don't think he's gouging too much - consider the insane CA tobacco taxes plus the added cost of doing business in CA plus the fact that since they are a rare cigar, he can gouge a lot more than that and still sell them.
Some of what I do at my B&M to get the best prices possible are the following:
1) I know the owner somewhat since I've bought a lot from him in the past. This helps a lot.
2) When you're going to buy something, try to buy a little larger quantity than you might want to. This increases their overall profits for the sale. No one wants to give any price break on 1 cigar, they're making a small enough profit already.
3) When you're almost done picking out your cigars and you have at least 4-5 premiums or $50+ total (see #2), you can do a couple things at this point. Pretend the calculator in your brain hasn't been furiously crunching away the whole time and ask him for the 'cash price' for the cigars. If it's a small shop and you're dealing with the owner or a manager, they may have the freedom to knock off a little tax for you - not to mention the 2-4% that would go to the Credit Card company on a charge.
4) Or, ask him if he'll throw in one of your sticks for free. My B&M sometimes will do a buy X get 1 free type deal.
Just remember to be respectful and be willing to bend the rules a bit. Usually I'll get a 1 free deal or at least 10% shaved off the price when I use these methods.
As a general rule, I try to avoid buying anything in-state but the rare cigars I can't get online. As a result, my B&M thinks I have a very cultured taste for expensive smokes

Makes it good though because he's always ready to flash me the Opus/Anejo/Padron 80th/etc. that he has stashed away when I walk in the door.
IMHO - If you call around and shop at many different B&Ms - it's harder to build the relationship that's gonna get you some good discounts.
Edit: to do a bit of rewording.