How do I get into the castle?

Listen to what the eye at the door (past the drawbridge) told you.
Everything you need to know can be found in what he says.

No matter what you do, you're not getting in through the front door.

But there is an exit elsewhere beyond the moat.

See the hint about crossing the moat, and take the alternate way as described
in that question.  You should be able to get into the sewers by following
that path.

The switches in the sewer are the key to getting in.

Remember that Xanth uses a lot of wordplay.

Substituting one word for another that sounds the same is often helpful.

Listen to the eye's "mark my words" speech again.  And mark his words.

Mark them by flipping the corresponding switches.

Say his speech out loud, and listen to yourself.  Ignore the printed words.

"For one to gain final entrance" contains the answer.

"Four one two gain final entrance" might be a better way of saying that.