The basics are that it was a common view that taxation on capital would be double taxation on saving leading to lower capital formation as well as being inequitable. This is true enough when one is thinking about the economy in the form of a simple two goods model with choices between consumption and investment where investment amounts to little more than deferred consumption. If there is also a present value discount of future consumption, this effect is even stronger. In this view, people largely save now because they believe they will be able to consume more later by doing so.
However, more recent literature is skeptical of the strong case in favor of consumption over capital taxation (though I am not aware of it bothering to give a critique of this simple accounting identity regarding taxing saving as double taxing income, nor should it really, economics is a study of human behavior not natural law so if behavior does not match the accounting identity it's the accounting identity that is wrong, not human behavior). To quote Batina and Ihori's Consumption Tax Policy and the Taxation of Capital Income:
That's a lot of caveats, all of them realistic. There are other problems with the idea of exempting capital income from taxation. For instance, while under the assumption that savings today is the result of expected higher, time discounted consumption tomorrow a capital tax will reduce investment, investment will be increased if I assume savings today is the result of a desired income level tomorrow (such as under a lifetime income hypothesis). With this assumption, if I want to save $1 million for my retirement I will have to save more today if there is a capital tax than if there is not, the capital tax now has the opposite impact that it did under a separate set of assumptions (though it is potentially still distortionary regarding utility measures). This is probably closer to how most savers in an economy behave.
It may be suboptimal to exempt capital from taxation if any of the following conditions hold: government spending is strictly proportional to output; precautionary saving arising from incomplete insurance markets exists; there is productive public spending that enhances private investment; there are liquidity constraints; and negative externalities associated with production exist. The general case for the consumption tax is very weak in the presence of any of these phenomena.
Other assumptions create similar complications. If we add a human capital term so that an individual has a choice between standard capital investment and human capital investment taxing the income derived from human capital in the form of increased wage income and not capital income derived from capital gains and dividends distorts this choice as well. Enter in bequest terms and the various bequest strategies that have been found in empirical studies of this behavior and things get even more complex.
Stepping outside of more formal economics and just thinking about savings behavior that we encounter as individuals throws more complications up. Many people just save excess income, if I have plenty of books on the shelf that I haven't read yet my savings account just gradually increases (assuming my girlfriend doesn't have some more exciting ways to spend money, I tend to be kinda boring). I only consider the return if enough money is sitting there that I think it would be wasteful not to invest, taxes don't even occur to me. Saving in case of a rainy day is rather common, things happen and we want money just in case. Also, it should be noted that studies of this find that alternatives to capital taxation, such as wage and consumption taxes, distort labor leisure trade-offs creating another source of distortions not always mentioned when capital taxes are discussed. There are strong hints that it is labor supply elasticities and not savings elasticities that are of more importance to capital taxation.
I could go on, but I think the general point is clear enough. The idea that taxing capital is double taxation is largely the product of the assumptions of certain models used to analyze this topic, use different assumptions and there are different results. This is something that I think should be emphasized more in undergraduate courses (or perhaps in economics generally, I know I heard this much more in my graduate poli-sci courses than I did in my undergrad economics). Models are excellent tools for thinking about relationships between the concepts under study. However, they are also prone to simply spitting your assumptions back out at you, if a simple two good economy involving consumption and investment is assumed then there will be one set of results, if instead human capital vs. physical capital is being modeled or labor market dynamics, different sets of results and policy recommendations will be the product of the model.
From a policy standpoint however, it is most likely the case that all assumptions are valid to one degree or another, some people will face decisions between human capital and physical capital others will follow a strict lifetime income model while others will make consumption decisions that conform more closely with consumption maximizing behavior. Which model to use then becomes primarily an empirical question, given current starting conditions which set of behaviors is most accurate for the polity under study. It is also likely that the answer will vary depending on both location and time, one polity may be capital short and need policies to maximize capital investment even at the expense of human capital another may have labor market problems and need a focus on human capital development. Dealing with the problems of each of these will require different conceptualizations of how income, savings, and capital are treated. There is no "right" way to think about these subjects, they are contingent on the situation and on what is being studied. Optimal policy requires the ability to approach a problem from multiple angles, otherwise disaster awaits.