How to append to a table in Lua

Your code is efficient but wrong. (Consider {[false]=0}.) The correct code is

if next(myTable) == nil then
   -- myTable is empty
end

For maximum efficiency you'll want to bind next to a local variable, e.g.,

...
local next = next 
...
... if next(...) ...

(When next is local, the code finds primitive function next by a constant-time indexing operation into an array of "upvalues." When next is left global, finding next involves indexing index the "environment" hash table, which contains the values of the global variables. This indexing operation is still constant-time, but it is significantly slower than the array lookup for a local variable.)

No, setting the key's value to nil is the accepted way of removing an item in the hashmap portion of a table. What you're doing is standard. However, I'd recommend not overriding table.remove() - for the array portion of a table, the default table.remove() functionality includes renumbering the indices, which your override would not do. If you do want to add your function to the table function set, then I'd probably name it something like table.removekey() or some such.