Python – Multiple sites with the same codebase in Python

Architecturedjangomemorypythonweb-applications

I am trying to run a large amount of sites which share about 90% of their code. They are simply designed to query an API and return the results. They will have a common userbase / database but will be configured slightly different and will have different CSS (perhaps even different templating).

My initial idea was to run them as separate applications with a common library but I have read about the sites framework which would allow them to run from a single instance of Django which may help to reduce memory usage.

https://docs.djangoproject.com/en/dev/ref/contrib/sites/

Is the site framework the right approach to a problem like this, and does it have real benefits over running separate applications?

Initially I thought it was, but now I think otherwise. I have heard the following:

Your SITE_ID is set in settings.py, so in order to have multiple
sites, you need multiple settings.py configurations, which means
multiple distinct processes/instances. You can of course share the
code base between them, but each site will need a dedicated worker /
WSGIDaemon to serve the site.

This effectively removes any benefit of running multiple sites under one hood, if each site needs a UWSGI instance running.

Alternative ideas of systems:

I don't know what route to be taking with this.

Best Answer

Why are you optimizing?
You seem to be concerned about memory usage. Why? Is the projected savings large enough to let you rent a cheaper VM? Is the sites' current usage threatening to bust your machine's current RAM? (And even if they are, are you confident that your going to spend less on development time for this feature than you would for just getting a better machine?)

Let's assume you already asked yourself that...
And the unfortunate conclusion was that there is a compelling reason to optimize memory usage.

You are talking about putting in work and code infrastructure to link these sites into a single, optimized unit. If these separate sites are inextricably linked to each other in some way other than sharing the same code base, it makes sense to build physical infrastructure on top of that logical grouping. If these are functionally separate sites that just happen to use mostly the same codebase, I would advise to not even think about using a framework like this unless you've got no other choice.

What if one of the sites later wants to pick up a windows dependency (assuming the others are running on *nix right now)? What if one of the sites becomes so popular that it needs to be on a separate machine for performance reasons? What if the customer wants to relocate one of the sites for better response times in Europe?

Without any other details about what you are working on, creating an architecture that ties these sites together sounds like a pretty questionable endeavor.

Folder Structure

your-project/
  apps/
    global.html
    app1/
      index.html
      app1.module.js
      app1.js
      parts/
        foo.js
        foo.html
        ...
    app2/
  libs
    lib1/
      lib1.module.js
      parts/
        directive1.js
        directive1.html
    lib2/
  third-party/

Configure your server web framework to find apps/app1/index.html when a request for /app1 comes in. Use friendly URLs (e.g. the-first-application/ instead of app1/ and map them using your server technology if required.
Your server technology must include global.html in index.html because it contains the vendor includes (see below).
Include the assets required by the respective app in index.html (see below).
Put the ng-app and the root <div ui-view></div> in the index.html.
Every app and lib is a separate angular module.
Every app gets a <app-name>.module.js file that contains the angular module definition and dependency list.
Every app gets a <app-name>.js file that contains the modules' config and run blocks, and the routing config as part of it.
Every app gets a parts folder that contains the applications' controllers, views, services and directives in a structure that makes sense for the specific app. I don't consider sub-folders like controllers/, views/ etc helpful, because they don't scale, but YMMV.
Libs follow the same structure as apps, but leave things out that they don't need (obviously).

Start out with services and directives within the app where they are used. As soon as you need something in another app as well, refactor to a library. Try to create functionally consistent libraries, not just all-directive or all-services libraries.

Assets

I minify both JS and CSS files for release builds but work with unminified files during development. Here's a setup that support this:

Manage vendor includes globally in global.html. This way, the heavy stuff can be loaded from cache when navigating between SPAs. Make sure caching works appropriately.
Per-SPA assets are defined in index.html. This should only include the app-specific files as well as the used libraries.

The folder structure above makes it easy to find the right files for the minification build steps.

Database – Splitting application into multiple but keeping database same

Microservices have their own set of challenges. What you are describing is essentially using the Strangler Pattern to replace the old application. This gives you some benefits:

You focus your efforts on one change so you can get the balance of functionality across your microservices correct
You keep the data store working which was working before

However, it does have some limitations:

Your bottleneck is now your data store
Scaling your application is limited to the database architecture

So, I would say it's an effective way to split the complexity of the changes you need to do. The question is whether you understand where your current bottlenecks are for your application.

If you are nowhere near the limits of your current database architecture, then I would power on with your approach as the initial cut. It allows you to focus on the complexity of getting your microservice infrastructure working together without worrying about how data is stored. You can leverage a lot of existing code that handles the database interaction.

That buys you time to consider what changes you need to make.

Measure your application so you understand the bottlenecks
Use doctrine (all the "rules" for microservices) as a guideline to evolve your architecture
Decide what changes are appropriate to address the rate of growth for your system.

In short it doesn't have to be perfect right off the bat. It just needs to be good enough to focus on what the next biggest problem is. And know that microservices bring with it a complexity you won't have in a monolithic application. However, they solve a problem that monolithic apps don't have--one of serving customers at internet scale.

Best Answer

Related Solutions

Design – AngularJS: structuring a web application with multiple ng-apps

Folder Structure

Assets

Database – Splitting application into multiple but keeping database same

Related Topic