Perspective taking—the ability to see things from someone else’s point of view—can boost success in communication. A signaler might take perspective when designing an utterance that is informative from the receiver’s point of view, or the receiver might take perspective when inferring the signaler’s communicative intentions. Perspective taking is supposed to play a particularly vital role when people try to communicate in the absence of a conventional signaling system. However, the task demands in such cases are extremely different from those in typical experimental approaches to perspective taking. Thus, current evidence for perspective taking does not establish whether humans can take perspective in those cases where perspective taking is arguably most helpful. We describe experimental tests of perspective taking that are suitable for settling the matter. Our task focuses on the use of shared world knowledge rather than shared visual scenes, and it is suitable for both open-ended and contextually constrained responses. We show that people generally fail at perspective taking in a novel signaling task, but that perspective taking can be boosted by contextual constraint. In that case, however, it is context, rather than perspective taking or shared world knowledge, that explains communicative success.