Here's how you solve getting the red crystal:
Drink the stew with the added mushroom to get strong; pick up the ton weight and walk across right hand 'moat reclamation' area to drop down and get the pickaxe; use the pickaxe on the brick wall in the brown castle; pull the exposed lever; go left and now see that a 'stone' moves up and down; jump on and jump right at the top to land on the roof and collect the red crystal.
The blue crystal is hidden in part of the foliage of the 'tree house' complex - just jump around till you can land on a branch.