A bedrock principle of CrossFit is that all the workouts we do and all the movements we use can be modified to suit anyone willing to put forth the effort, regardless of age or previous physical condition. Take a look at this video on how a diverse group, which includes a woman over 60 years old, all do the same workout scaled to each individual's abilities, Scaled Workout [mov][wmv]. This is what we mean when we say we scale load and intensity.
Much of our coaching expertise centers around teaching people to move correctly, and guiding them through exercise progressions suited to their abilities. The modifications we make preserve the exercise stimulus from the Workout of the Day. Can't do a pushup? No problem, we have a substitute movement. Can't do a pullup? No problem, we have a substitute movement. Can't squat? We can fix it.
If women over 50 with little or no fitness or athletic training can do CrossFit, then so can you. Still dubious about it being too hard? Read the Mary Conover Story, "A CrossFit Grandma"