.net – Multi-threading in IronPython

ironpythonmultithreadingnetthread-safety

I have a "script class" in IronPython, and scripting in my app works by calling methods on its instance. I need to implement calling scripts from multiple threads. What is the correct way to do it?

I have multiple concerns:

Is ScriptScope thread-safe? Information is contradictory. ScriptScope's documentation says: "ScriptScope is not thread safe. Host should either lock when multiple threads could access the same module or should make a copy for each thread." However, IronRuby uses the same DLR and @JimmySchementi says that "ScriptRuntime, ScriptEngine, and ScriptScope are all thread safe, designed to be used between threads. Specifically, ScriptScope uses a thread-safe data-store, so ScriptScope can be shared between threads."
If I create multiple ScriptScopes, that would mean executing the same initialization script multiple times. Let's suppose that I run ten Python script files, import five assemblies and on the whole execute quite a bit of code to get the "script object" ready. Is there any way to avoid the time and memory cost of running a lot of the same code for each and every thread?
Is making the ScriptScope variable thread-static (that is, applying ThreadStaticAttribute) and executing initialization for every thread which is utilized by Task.Run the way to go? Or should I use a TaskScheduler with a limit on concurrency, because the cost of multiple scopes is high?

On the whole: how to correctly implement running the same script on different arguments in multiple threads? Scripts must be executing simultaneously and must not crash due to race conditions.

Best Answer

1.

If the documentation of ScriptScope says it is not thread safe, believe it, or at least act like you believe it. @JimmySchementi may have looked at the current implementation and worked out that it is currently thread safe, but this give no guarantees about how it will behave in the next patch of the class, let alone the next major release.

2.

Yes you will need to initialise each of your ScriptScopes. I would try and minimise the number of ScriptScopes you require, how to do this will depend on your set up. If the main purpose of the threads concerned is to host a ScriptScope then you should use a ThreadPool with each thread having one ThreadLocal<ScriptScope>. If these threads are doing other things as well as running scripts then you should have an Object pool storing the ScriptScopes and each thread can checkout the ScriptScopes, do the work, then release the ScriptScope.

3.

Prefer ThreadLocal over ThreadStatic if you go down this path.

Multi-threading is possible in php

Yes you can do multi-threading in PHP with pthreads

From the PHP documentation:

pthreads is an object-orientated API that provides all of the tools needed for multi-threading in PHP. PHP applications can create, read, write, execute and synchronize with Threads, Workers and Threaded objects.

Warning: The pthreads extension cannot be used in a web server environment. Threading in PHP should therefore remain to CLI-based applications only.

Simple Test

#!/usr/bin/php
<?php
class AsyncOperation extends Thread {

    public function __construct($arg) {
        $this->arg = $arg;
    }

    public function run() {
        if ($this->arg) {
            $sleep = mt_rand(1, 10);
            printf('%s: %s  -start -sleeps %d' . "\n", date("g:i:sa"), $this->arg, $sleep);
            sleep($sleep);
            printf('%s: %s  -finish' . "\n", date("g:i:sa"), $this->arg);
        }
    }
}

// Create a array
$stack = array();

//Initiate Multiple Thread
foreach ( range("A", "D") as $i ) {
    $stack[] = new AsyncOperation($i);
}

// Start The Threads
foreach ( $stack as $t ) {
    $t->start();
}

?>

First Run

12:00:06pm:     A  -start -sleeps 5
12:00:06pm:     B  -start -sleeps 3
12:00:06pm:     C  -start -sleeps 10
12:00:06pm:     D  -start -sleeps 2
12:00:08pm:     D  -finish
12:00:09pm:     B  -finish
12:00:11pm:     A  -finish
12:00:16pm:     C  -finish

Second Run

12:01:36pm:     A  -start -sleeps 6
12:01:36pm:     B  -start -sleeps 1
12:01:36pm:     C  -start -sleeps 2
12:01:36pm:     D  -start -sleeps 1
12:01:37pm:     B  -finish
12:01:37pm:     D  -finish
12:01:38pm:     C  -finish
12:01:42pm:     A  -finish

Real World Example

error_reporting(E_ALL);
class AsyncWebRequest extends Thread {
    public $url;
    public $data;

    public function __construct($url) {
        $this->url = $url;
    }

    public function run() {
        if (($url = $this->url)) {
            /*
             * If a large amount of data is being requested, you might want to
             * fsockopen and read using usleep in between reads
             */
            $this->data = file_get_contents($url);
        } else
            printf("Thread #%lu was not provided a URL\n", $this->getThreadId());
    }
}

$t = microtime(true);
$g = new AsyncWebRequest(sprintf("http://www.google.com/?q=%s", rand() * 10));
/* starting synchronization */
if ($g->start()) {
    printf("Request took %f seconds to start ", microtime(true) - $t);
    while ( $g->isRunning() ) {
        echo ".";
        usleep(100);
    }
    if ($g->join()) {
        printf(" and %f seconds to finish receiving %d bytes\n", microtime(true) - $t, strlen($g->data));
    } else
        printf(" and %f seconds to finish, request failed\n", microtime(true) - $t);
}

Python – IronPython memory usage

You could try creating a new AppDomain every time you run one of your IronPython scripts. Although assebmlies cannot be unloaded from memory you can unload an AppDomain and this will allow you to get the injected assembly out of memory.

Best Answer

1.

2.

3.

Related Solutions

Php – How to one use multi threading in PHP applications

Multi-threading is possible in php

Python – IronPython memory usage

Related Topic