You may be surprised to learn that this question has generated a great deal of debate among economists. One way of approaching the question was by applying the Baumol-Tobin model of cash management, which takes into account several variables.
Professor Greg Mankiw ran the model for a person who spends $10 a day of cash, earns $60 an hour, takes 10 minutes per ATM visit, and makes 5% annual interest on his bank account. For this person, the model shows that this person should take out $1,200 three times a year, carrying an average balance in his wallet of $600. Much of this surprisingly high number is a function of the value of this person's time.
In 1995, Mankiw calculated that even the average American would be better off carrying an average of $551.05, when in fact the average citizen carried around $75.
A new study by Greg Hannsgen of Bard College injects an obvious complication not included in Baumol-Tobin; the potential for a criminal to relieve us of our pocket cash.
Using a plausible set of parameters to represent the impact of crime on our pocket money drops the ideal number, as does the expectation that people who carry large amounts of money attract thieves at a greater rate that those who don't. He also accounts for people's fear that they might experience injury to themselves or their property during the theft.
Even taking these factors into account, though, Hannsgen concludes that the ideal average amount of pocket cash drops only from $750.05 to $516.48. Since in 1995, the average American carried only $75.48, he states that that "Americans spent over 1.4 billion extra hours going to the bank in 1995 because of fear of crime."
This assumes, of course, that I have $500 in the bank, which would be nice.