I have been playing with Custom Authentication lately and while trying out, I ran into bunch of problem. So I thought I will point out few of things you need to watch out. When you create a SL3 + RIA services, it automatically gives you Windows Authentication. When you need to change to custom authentication, couple of things you need to change and for that I would recommend Brad Abrams (ex-Microsoft) blog
If you are new to this area I would strongly recommend you to look at the authentication quick starts in the following URL, they are very simple and easy to follow.
while you are at it, I would recommend visiting this Silverlight forum question to have an understanding of authentication as well.
Few things to watch out are
1. Make sure you have <authentication mode="Forms" /> in your web.config at the WCF service side.
2. The USER return from GetAuthenticatedUser has a catch, if you do not assign the name to the returning user, then by default IsAuthenticated flag will be false.
3. Make sure you have authentication setup to form authentication in silver light app.xaml.
4. If you have a page where you are authenticating and then data binding, do not do data binding till you complete the validation. In other words, make sure you add the root visual element on login completed method delegate.
If you run into any other problems let me know or if you have any comments or suggestion also send me a mail.