We definitely save money shopping at Costco on a regular basis, both in the cheaper price for buying in bulk, plus the cash back rewards from using their credit card (pay off each month).
There are food staples that are generally cheaper, gasoline is always the lowest price in my area, and if you keep your eye open there are deals to be had on clothing and home furnishings. I think that their produce is very difficult to beat in terms of quality, let alone the price, for the most part, and the cheese I like is significantly cheaper. There's probably ways to get stuff cheaper by religiously comparing prices and cutting coupons for everything, but I have no patience for that.
I avoid the electronics section, as a general rule, unless there is something specific I need, because there's too much temptation to buy crap I don't need and could become obsolete before I get around to using it. And since it's been about a decade since they carried decent tools on a consistent basis, I don't blow much money on that aisle anymore, either.
My biggest complaint with Costco is the crowds and seemingly deliberate method of positioning demos and food samples in order to slow navigation through the store. I heard a podcast awhile ago that talked about all the analysis that goes into figuring out all the things that makes people put more crap into their carts before they hit the checkout lane. So you do need to be mindful that, to a certain extent, we're all being played when we step inside.