That's a reductionist argument. By that I mean generosity is seen in some absolute terms.
There are the haves and the have nots. The former are some how extra privileged to be sanctified enough in providing for the later. In this case you label non-politicised giving as those extra priveleged. I suppose that wealthy conservatives historically and moderm business titans are some how better philanthropists by default.
The problem with your meme is the most crucial function of giving is kindness psychologically. That is a non-reductionist sentiment most would agree upon. Otherwise, while some giving is foolhardy, there arises a litmus test for kindness. In that case the whole world becomes unkind.