If you have trouble balancing on the 2x4 on the ground due to lack of skill, you'll still have trouble at 10 feet even if you are entirely comfortable with the height.
If you are comfortable on the ground, you'll be comfortable at 6 inches. After awhile, you won't notice the 6 inches at all... and you'll be comfortable at 8, then ten, then a foot, and so on, without ever exceeding that comfort zone limit. Improvement will have come through experience, not because the board got jacked up to where you were scared. The comfort zone simply gets bigger.
A quick Google search suggests that this is in fact how gymnasts train for the balance beam. They even make graduated-height beams, so that once the skills are mastered closer to the ground (because learning happens when you're not scared), the gymnast moves up in height. Observe:
The saying I like is that you "sneak up on fast."
Last weekend for one session I bumped down to novice, partly because I was getting tired and partly to ride with some friends. I was markedly (even surprisingly) faster than the riders in that group, comfortable passing all over the place, etc.
Clearly, I've gotten a lot quicker since I was a novice-group rider. Yet I've never really tried to push out of my comfort zone. Never crashed. Never had a truly close call (a few "moments" and agricultural excursions, sure, but minor).
If improvement only comes from pushing beyond my limits, then how did that happen? How did I get faster? (Again, I think this is really just semantics... we appear to be defining "limits" differently.)
I do agree that you'll progress much more quickly and in bigger steps if you get a sink-or-swim kick in the butt, or if you consciously try to push the envelope.
But if my personal experience is any indication, you'll still get there even if you don't. The difference is that if you push or get pushed before you're ready, the risk goes way up.