You need to change the RichTextCtrl.ascx file (under issue) and add the MaxImageSize setting:
<radE:RadEditor id="RadEditor1" Runat="server" ToolsFile="~/RadControls/ToolsFile.xml" Scheme="Office2000"
ShowSubmitCancelButtons="False" Editable="True" TemplatePaths="~/issue/RAD/UserTemplates"
UploadTemplatePaths="~/issue/RAD/UserTemplates" DeleteTemplatePaths="~/issue/RAD/UserTemplates"
UploadImagesPaths="~/issue/RAD/UserImages" ImagesPaths="~/issue/RAD/UserImages" UploadMediaPaths="~/issue/RAD/UserImages"
MediaPaths="~/issue/RAD/UserImages" FlashPaths="~/issue/RAD/UserImages" UploadFlashPaths="~/issue/RAD/UserImages"
UploadDocumentsPaths="~/issue/RAD/UserImages" DocumentsPaths="~/issue/RAD/UserImages" AllowThumbGeneration="True" MaxImageSize="1048576"></radE:RadEditor>
This example will give you up to 1MB.