People are poor at assessing their own memory
One thing research seems to show rather consistently is that, for older adults in particular, beliefs about one's own memory performance have little to do with one's actual memory performance¹. People who believe they have a poor memory are usually no worse at remembering than those who believe they have a good memory.
One theory for why this might be, is that people may be influenced by their general beliefs about how memory changes with age. If you believe that your memory will get progressively and noticeably worse as you get older, then you will pay more attention to your memory failures, and each one will reinforce your belief that your memory is indeed (as expected) getting worse.
Memory decline can be a self-fulfilling prophecy
Research has also shown that common, everyday memory failures tend to be judged more harshly when the failure belongs to an older adult². What is laughed off in a younger adult is treated as an indication of cognitive decline in an older person.
There are ways in which cognitive function (memory, reasoning, problem-solving, etc) declines with age, but it would be fair to say that general belief over-estimates the extent of this. It is, to a large extent, a self-fulfilling prophecy. If you believe deterioration is inevitable, you are not likely to make any effort to halt it.
Memory decline is associated with physical factors
A large-scale study that tracked seniors over a ten-year period found that cognitive decline is not a normal part of aging for most elderly people: 70% of the nearly 6000 seniors in the study showed no significant decline in cognitive function over the ten-year period. These people had two factors in common: they did not carry any of the apolipoprotein E4 genes (often associated with Alzheimer's disease), and they had little or no signs of diabetes or atherosclerosis³.Other factors that have also been implicated in age-related cognitive decline are obesity, smoking, and high blood pressure. Indeed, researchers have suggested that risk factors for cardiovascular disease are also risk factors for cognitive decline: what's bad for the heart is also bad for the brain.