R – Silverlight WebPart in SharePoint

I'm making a WebPart for SharePoint that will instantiate a Silverlight UserControl and feed it some data. My problem is that when I have created my sample-WebPart and just instantiate a Silverlight control, the webpart, when added to a page or displayed in the webpart gallery, instead of rendering blank, renders an error page saying "File Not Found". No clue in the logfiles to what file was not found or why this error is thrown. Here is my code:

using System.Web.UI.WebControls;
using System.Web.UI.WebControls.WebParts;
using System.Web.UI.SilverlightControls;

namespace TestSLWP {
  public class CustomWebPart1 : WebPart {

    protected override void CreateChildControls() {
      Label lblHello = new Label();
      lblHello.Text = "Hello";
      Controls.Add(lblHello);
      Silverlight sl = new Silverlight();
    }
  }
}

I've added references to System.Web.Extensions and System.Web.Silverlight to the project. They are in the GAC, and the webpart is written and compiled on the same computer that SharePoint resides. If I change the CreateChildControls() to be:

protected override void CreateChildControls() {
  Silverlight sl = new Silverlight();
  sl.ID = "CustomWebPart1SL";
  sl.Source = "/Silverlight/CustomWebPart.xap";
  this.Controls.Add(sl);
}

I get the same error. Also if I remove the first slash in sl.Source I get the same error, even though the file is present in a virtual directory in the same application pool as SharePoint. I therefore, and because the error comes with just instantiating the Silverlight object, believe that the file that cannot be found is not my XAP.

What file can't SharePoint find and what can I do about it?

Here's the error message:

http://www.freeimagehosting.net/uploads/2dca8dbdfb.png