Short answer

Instead of using a custom function use a built-in function when possible or use a function triggered by an event.

Explanation

As the OP mentioned, Google Sheets has a built-in function to return a random numbers between 0 and 1: RAND.

In cases where it's required to do some complex calculations that are not possible or efficient with built-in functions, one alternative is to use Google Apps Script. It's worth to bear in mind that custom functions are recalculated only when one of their arguments changes, by the other side their arguments should be deterministic, so using functions like NOW(), TODAY(), RAND() and RANDBETWEEN() will return an error even if they are nested in other functions spreadsheet functions or included indirectly through references.

Google Apps Scripts has two kind of events, simple and installable. See https://developers.google.com/apps-script/guides/triggers/ for details.

Example

The following code will update the specified single cell reference every time time that an user edits the spreadsheet with a random number generated by a the Math.random().

var reference = 'A1';

function onEdit(e) {
  SpreadsheetApp
    .getActiveSheet()
    .getRange(reference)
    .setValue(Math.random());
}

References

• Custom Functions in Google Sheets - Google Developers
• Issue 246: Script function does not recalculate - google-apps-script-issues
• Refresh data retrieved by a custom function in google spreadsheet - Stack Overflow

The particular issue was sorted out in comments, but to have a generally useful answer here I include a helper function for checking whether an array is rectangular. If .setValues(something) throws an error, put checkArray(something) before that line. The function checks whether the argument is a double array in which all rows have equal length. Its output is found in the log, viewable by pressing Ctrl-Enter after running a script.

function checkArray(ar) {
  if (ar.constructor !== Array) {
    Logger.log('Not an array'); return;
  }  
  var height = ar.length; 
  if (ar[0].constructor !== Array) {
    Logger.log('Not a double array'); return;
  }
  var width = ar[0].length;
  for (var i = 0; i < height; i++) {
    if (ar[i].length !== width) {
      Logger.log('Not rectangular: row '+(i+1)+' has length '+ar[i].length+' instead of '+width); 
      return;
    }
  }
  Logger.log('Rectangular of height '+height+' and width '+width);
}

It would be great if Apps Script itself logged more detailed error messages of this sort, instead of just saying "Cannot convert Array to Object[][]".