Named pipes and TCIP protocols for SQL server 2005 are disabled by default. Have you enabled them under "SQL server configuration manager"? You can find the protocols in the SQL Server Network configuration and under SQL Native client xx configuration.
The connection on the server itself works thanks to the "Shared memory" protocol.
You may want to look at FmPro Migrator. It will convert FileMaker Pro (2,3,4,5,6,7,8,9,10) to a SQL Database (MySQL, Oracle, Access, Microsoft SQL Server, Sybase, DB2, OpenBase, PostgreSQL, FrontBase, SQLite and Valentina).
You could also FileMaker Instant Web Publishing. It's part of FileMaker Server Advanced and is also part of the client FileMaker. It's quite easy to use as it allows you to design your layouts in FileMaker and lets your users access the database via a web browser.
You can also access the data via ODBC from your other database or program.
Lastly, you can export FileMaker's data from within the FileMaker client program to an XML or text format and the import into your new database format and build new forms and layouts.
Best Answer
Unfortunately, ODBC is is the only game in town. Currently there is no better way in the market that allows you to do it without ODBC, nevertheless the installation process can be complicated for non-programmers.