API Development – Writing a Language Agnostic API

apicdesktop applicationlanguage-agnostic

I'm planning on writing a C++ program which is heavily influenced by plugin modules. Initially I had only thought of making the APi available as an abstract class that others extend. After talking to someone they mentioned that I should think about making it available in a language agnostic way.

How do I effectively expose an API in a manner that any language can use it?

The solution that I could come up with was having the C++ API like I originally intended, for C++ directly and CLI languages (C#, F, etc) and then a sort of document and executable combination. Where expected input is described in the document and then the executable is started (via console/terminal) I would give parameters given as start up commands. This would theoretically work but doesn't sound like a standard solution too me. How is this done in other desktop software?

Best Answer

You cannot do that because some languages implementations could have an ABI different and incompatible with the C one.

On current systems, C is very common and most (but not all) language implementations have an ABI and calling conventions compatible with the one from C. Notice that it is a property of the implementation (your C compiler and your operating system), not of the programming language.

Also, some languages have different and incompatible views on control flow (look into call/cc and tail-calls in Scheme, Goroutines in Go, or backtracking and cut operator in Prolog, CLIPS rules, concurrent actor languages, ....). I'm not sure you'll be able to design something which easily fits all of them (and that could impact the FFI requirements of implementation of these languages).

If using C++, beware of name mangling and of exception handling. (e.g. C longjmp is not friendly with C++ exceptions). Dynamic loading facilities like dlopen and dlsym are relevant to name mangling. So prefer an API using extern "C" functions. libgccjit could be inspirational (it is coded in C++ but has a C friendly API) and perhaps useful (you could consider runtime generation of glue code).

Memory management (notably with garbage collection) is also an issue. Study for examples foreign function interface of Ocaml and of SBCL and of Lua and of Guile. Look into libffi. Some languages also want serialization or persistence facilities (even for foreign data). Read also about dynamic software updating.

You could provide some reflection facilities (e.g an API to query your API, e.g. like GTK introspection). You might try to provide a generic closure mechanism like in GObject-s.

You could use (or customize or adapt) code generators like SWIG. You might consider compiler plugins (e.g. GCC MELT extensions).

If you have a wide and complex API (hundreds of public functions or data types) consider at least providing a machine readable form of it, e.g. some database or XML or JSON from which automatic glue code could be generated.

There is no silver bullet.

NB. Better make your stuff free software. You might get outside contributions and you'll need outside help to interface your thing with many programming languages, including some that you don't even know. Open source is a good way to counter leaky abstractions (since you and other contributors can dive into the implementation source code). Given the variety of programming languages, you will need outside help (as soon as your API is successful)


why the question don't make any sense

I asked several times the OP to give much more context and motivation, and to explain the domain for which the library is written, and the context in which that library would be developped. Sadly I've got no answers at all.

The applicative domain (e.g. high speed frequency trading, software for dentists, selling machine, weather forecast, image processing, speech recognition, word processing, social network, static source code analysis ...) matters practically a lot, because it defines what programming languages are likely to be considered and what programming paradigm is probably used. For instance, an image processing library is very unlikely to need to be interfaced to Agda or Coq, but probably would be used from C or C++ applications. Business software is often written in Java (and older ones was in Cobol) so needs to be interfacable to JNI or JVM. Static source code analysis software is very likely to use some theorem prover (so Ocaml or Common Lisp or Haskell is important to them, and they need to be garbage-collection friendly). And so on. Sometimes, inter-process communication (including RPC, MPI, web services, RESTful applications) can be used, but at other times it is not efficient or not convenient enough. Details and context are very important and we don't know them.

The question mentions "desktop software", but that just means today something having a native (non-Web) GUI used with a mouse and a large enough color screen (the dentist software, the software for managing my bank accounts, your favorite game software, the word processor, a web browser are all desktop software, but their main commonality is just having a GUI). Using Qt (a very powerful GUI cross-platform toolkit for C++) facilitates the development of such things. And quite often a library used on desktop (think of XML or JSON libraries like Xerces and JsonCPP, machine learning libraries like TensorFlow, HTTP client libraries like libcurl, numerical computation libraries like BLAS or GMPlib, etc...) don't even care about GUI (but it could care about response time -having functions returning in less than 0.4 seconds- and thread friendliness), because the application (not a reusable library) would care about GUI itself.

The question mentions plugins, but these are simply compiled software components which get dynamically loaded at runtime (thus increasing the virtual address space of the process loading them), using e.g. dlopen on POSIX (and probably LoadLibrary on Windows, which I don't know at all). Notice that the JVM don't use (Java-coded) plugins but dynamically loaded classes, and speaking of plugins don't make sense with JavaScript or with Common Lisp (or most languages having a compile or eval primitive). So the mention of plugins don't means much (both Firefox browser and GCC compiler accept plugins, for very different reasons).

Without a lot more additional details (including applicative domain, context, motivation ...) the current question don't make any sense. I offer to delete my answer if that helps in closing the question in its current form.

PS. I don't know and never used Windows (but I use Unix since 1987, and Linux since 1993), but that is an unimportant detail.