How do you customize a character and avoid optimization?
Optimization requires several factors. Perfect accessible knowledge of the rules, an objective metric of value to maximize, and choice without rational consequence.
Unbalance one of these key factors and you should see much more randomization.
Clearly, if we are playing a game, we need to have objective metrics of value so that decisions can be made interesting. So in order to prevent people from simply making sure that they 'optimize' those values we can do several things.
First, make sure each choice has a 'rational consequence'. The biggest offenders of optimization gone wild in 3.5 is where you can take a thing of little value, like turn attempts per day, or any level past 2 of ranger, and trade them in for something of high value. Those choices do not have rational consequences, in that the consequence of the choice is not equal to the benefit.
The second thing that's a problem is having perfect accessible knowledge of the rules. This does not mean that the player can't look at the rulebook. What this means, is that if all options are presented and are limited to just those options then you end up with a situation where the player can just 'pick' the best option.
It is better to provide a system of option creation, that allows players to instead state what sort of in game effect they are looking to provide, and then allowing the player and the DM to come to terms with such an effect.
This way, instead of having 'feats' and 'classes' and 'powers' and 'spells' that you have to buy, you just have the rules they used to create such things. I would purchase such a thing, and players would play it, and character customization without optimization would be had!