The mechanisms and functional anatomy underlying the early stages of speech perception are still not well understood. One way to investigate the cognitive and neural underpinnings of speech perception is by investigating patients with speech perception deficits but with preserved ability in other domains of language. One such case is reported here: patient NL shows highly impaired speech perception despite normal hearing ability and preserved semantic knowledge, speaking, and reading ability, and is thus classified as a case of pure word deafness (PWD). NL has a left temporoparietal lesion without right hemisphere damage and DTI imaging suggests that he has preserved cross-hemispheric connectivity, arguing against an account of PWD as a disconnection of left lateralized language areas from auditory input. Two experiments investigated whether NL’s speech perception deficit could instead result from an underlying problem with rapid temporal processing. Experiment 1 showed that NL has particular difficulty discriminating sounds that differ in terms of rapid temporal changes, be they speech or non-speech sounds. Experiment 2 employed an intensive training program designed to improve rapid temporal processing in language impaired children (Fast ForWord; Scientific Learning Corporation, Oakland, CA) and found that NL was able to improve his ability to discriminate rapid temporal differences in non-speech sounds, but not in speech sounds. Overall, these data suggest that patients with unilateral PWD may, in fact, have a deficit in (left lateralized) temporal processing ability, however they also show that a rapid temporal processing deficit is, by itself, unable to account for this patient’s speech perception deficit.