Naturalism in the philosophy of mathematics was originally proposed by Quine and Putnam, its consequences have been thoroughly explored by Penelope Maddy, through a methodological reflection on the most recent developments in set theory. We argue that the crucial point in the debate on naturalism in mathematics is the relationship between naturalism and realism. Does mathematical naturalism presuppose a sort of metaphysical realism? If the answer is yes, any naturalistic philosophy of mathematics is in deep trouble. We show how the naturalistic view is apparently in need of realistic assumptions which are unacceptable from a naturalistic point of view. Starting from a survey of indispensability arguments, we point out their untentability, then follow Maddy in her anti-realist modification of naturalism, raise objections to this new position, and finally show that the only apparent way out is to go back to some form of realism, which is inconsistent with naturalism. Throughout the paper we test out naturalism by examining arguments for or against strong hypotheses in set theory.