A phoneme (such as /a/) is an abstract entity under which several different phones (actual speech sounds, such as [ä], [ɐ], [ɑ], [ʌ]) are classified in a speaker's mind. These differing phones are called allophones of one another (or of the phoneme).
There are two subtypes of allophones:
Continuous (phonetic) allophones
The human articulatory organs (and likely those of any physical being) are not perfect in operation. Since features such as the exact position of the articulators, intensity of phonation, the (speaker-dependant) shape of the oral cavity, and amount of food stuck between the teeth are continuous variables, any two attempts to articulate a sound will be ever so slightly different. Acoustics of the environment have further effect on the exact sound reaching the ears of an addressee. Yet, the brain ignores all this variation and recognizes a specific sound (this is proven to involve the recognition of specific formants' position and strength in the sound spectrum).
What exactly constitutes a "specific sound" is to some extent language-dependant. Two sounds that will be identical to one person's ear (say, [ɜ] and [ə]) will be typically recognized as distinct by someone speaking a language where the sounds are contrastive. However, not even this is a hard-and-fast division: depending on the degree of contrastive load, hearing or pronouncing a phoneme beyond its usual degree of variation may not matter, if context can be relied on to decipher the word.
Infants, it seems, will recognize a wide range of sounds, but will lern during language acquisition what sub-range of sounds to group under a given phoneme. The total variety seen in natural languages serves as a (loose) minimum indicator of what human hearing and articulation can possibly reliably distinguish.
There are no special IPA symbols for marking most continuous allophones, and the only exact way to record them is a recording of the sound itself; however, for a first approximation, diacritics such as raised/fronted/lowered/backed can be used.
Discontinuous (systematic) allophones
It commonly occurs that a phoneme's allophones fall in two — or sometimes more — groups that are separated, ie. given some arbitrary dimension, certain intermediate realizations occur rarely, if at all. There are a number of possibilities for this.
Featural phonology tells that a phoneme is not an atomic, indivisible whole: they are formed of features, such as [+nasal] and [+labial]. Under this view, most discontinuous allophones can be described as differing with respect to some feature. Taking the common example of [ŋ] as an allophone of /n/ before velar consonants, this allophone is distinct by having the feature [+velar] (or [+high], or [+back], depending), not included in the phoneme's own specification but inherited from the adjoining velar.
Sound changes account for another class of discontinuous allophones. A sound's exact realization usually depends on what other sounds are pronounced before and after. It can be that eg. a velar consonant is influenced by a succeeding front vowel such that it becomes articulated more front, approaching the territory of palatal consonants. This kind of a change can begin as a small extension of the phoneme's normal range of variation, that then detaches — for example, by assibilation, leading to a coronal affricate. If the palatal allophones did not occur elsewhere, the phoneme's allophonic range has now split in two groups. The results can persist as a purely phonetical variation, especially if the resulting sound does not particularly resemble other phonemes existing in the language; however, if it does, or contains an articulation quite distinct from the "normal" allophones, it is likely that the divergent allophones will gain a new featural analysis. In our exemplary case it is likely that [tʃ] as an allophone of [k] would cease to be thought of as containing the feature [+velar], since the articulation no more involves the velar gesture, and that the language in all likelihood does contain other coronal consonants.
Speakers can in some cases, tho not all, tell apart two allophones that are phonologically distinct. Articulatory distinctions can lead to even complete split, where the two sounds are no more even mentally associated together under the same phoneme. A possible example is the concept of heng: English /h/ and /ŋ/ are in fact in complementary distribution. Yet they are thought of as distinct phonemes.
The IPA is by and large designed to accommodate all discontinuous allophones found in human languages. For example, from an articulatory point of view the division of coronal consonants to dental, alveolar, postalveolar and retroflex is arbitrary; but since no natural language divides this continuum to more than four contrastive places of articulation, it is quite convenient phonologically. (It can then turn out unclear if a 2-part division should be described as between eg. dental and alveolar, or dental and postalveolar, but this rarely matters.)