[Home] [Downloads] [Search] [Help/forum]


Register forum user name Search FAQ

Gammon Forum

[Folder]  Entire forum
-> [Folder]  MUSHclient
. -> [Folder]  Python
. . -> [Subject]  Python co-routines, how to use them?

Python co-routines, how to use them?

It is now over 60 days since the last post. This thread is closed.     [Refresh] Refresh page


Pages: 1 2  3  

Posted by Worstje   Netherlands  (899 posts)  [Biography] bio
Date Sun 28 Oct 2007 02:30 PM (UTC)
Message
Simply put, does anyone have any experience with this? Especially in MUSHclient?

I've read up on the topic a bit and they seem like the solution I have - a lengthy process (goes beyond 5s+) that is capable of returning intermediate results. Since it is lengthy, I don't want it to block my triggers with a simple execution. Rather, I want it to process in the background while my triggers fire, and if a trigger needs one of those intermediate results, it can just request it.

The entire topic is new to me, so I'm kind of lost thus far.. any help people could offer is appreciated. I've been reading up on co-routines, but most uses of it are complicated in the sense that I'm not sure what it is achieving. :/

Thanks in advance! :)
[Go to top] top

Posted by Isthiriel   (113 posts)  [Biography] bio
Date Reply #1 on Wed 07 Nov 2007 11:56 AM (UTC)
Message
Co-routines are cooperative multitasking, nothing really goes on in the background, they share-and-share-alike the foreground.

The Python Help file has some information and PEP 342 has more (http://www.python.org/dev/peps/pep-0342/).

When Nick was talking about how to do coroutines in Lua, I posted the equivalent Python code (http://www.gammon.com.au/forum/bbshowpost.php?id=7511&page=2).

I think what you really want though for background processing are true threads. You can launch a new thread in Python fairly easily using the threading module which also has synchronization classes, so the intermediate (and final) results can be stored in Python variables available to the triggers.

The only important proviso is that the world object becomes invalid around the time the main thread returns from the script so you shouldn't try and use it in the second thread.

It's kind of late at night atm so if this doesn't make sense I will produce demonstration code after I've gotten some sleep :)
[Go to top] top

Posted by Shadowfyr   USA  (1,786 posts)  [Biography] bio
Date Reply #2 on Wed 07 Nov 2007 04:41 PM (UTC)
Message
Actually, in theory their should be a way around the world object being lost. You simply create another instance of it in the other thread. Mind you, that may not be as simple as making a new object type variable = world, it might require setting up a wrapper from the linker table used by external applications (i.e. mushclient.tlb), then connecting to the client via that. It kind of depends on how Mushclient hands over the "world" functions. If they are part of a class/object, then making a copy of the object before the main script exits should means that you still have access to them, from that copy. I have no idea if that will work though, or for that matter, if there is a way in Windows to make it *not* work. At worst, trying to do something like:

new object myhdl
myhdl = world

Or what ever the commands are in Python to create an empty object container, then copy an existing one into it, will simply fail.
[Go to top] top

Posted by Isthiriel   (113 posts)  [Biography] bio
Date Reply #3 on Fri 09 Nov 2007 10:54 PM (UTC)
Message
The problem is that you can copy the Python side of the world object, but because it's COM you don't have access to the C++ side of it and Something happens after execution passes back to MUSHClient that causes at least part of the world object to be invalidated... so some things work, some cause MUSHClient to segfault :P

Since I haven't made a list of what works and what doesn't (and under what circumstances that might change), I try and avoid doing anything with it. As it happens there is no great need to as there are work arounds.

http://www.gammon.com.au/forum/bbshowpost.php?id=7666&page=5

That example is built around having a Tkinter event loop as the second thread but is easily generalized.

I've since discovered the memory leak mentioned in there is avoidable if you call the global poll() function as a timer function rather than using Send-To-Script.

ie:
def poll(name=None):
    global apps
    for app in apps:
        app.poll()

[Go to top] top

Posted by Isthiriel   (113 posts)  [Biography] bio
Date Reply #4 on Sat 10 Nov 2007 06:10 AM (UTC)
Message
<timers>
  <timer script="TimerPoll" enabled="y" second="0.10" send_to="12" active_closed="y" >
  </timer>
</timers>



## I use Python 2.5.1 which has the with statement in __future__
## Python < 2.5 needs to replace the:
##     with <lock>:
##         <code>
## blocks with:
##     try:
##         <lock>.acquire()
##         <code>
##     finally:
##         <lock>.release()
## The following line must be before any other Python statements in the file:
from __future__ import with_statement

import threading
_qcode = []
_qlock = threading.RLock()

def TimerPoll(name):
    global _qcode, _qlock
    while len(_qcode):
        with _qlock:
            code, locals = _qcode.pop(0)
        exec code in globals(), locals

def TimerQueue(code, locals = {}):
    global _qcode, _qlock
    with _qlock:
        _qcode.append((code, locals))


Then we can convert from:
def some_function_with_lots_of_processing():
    data = []
    world.Note("Started!")
    time.sleep(2.0) # busy, busy
    update_interim_results(data)
    time.sleep(3.0) # so much hard work
    update_final_results(data)

def update_interim_results():
    world.Note("Interim Results Available!")

def update_final_results():
    world.Note("Job's Done!")


To:

def actual_sfwlop():
    data = []
    TimerQueue("world.Note(\"Started!\")")
    time.sleep(2.0) # busy, busy
    TimerQueue("update_interim_results(data)", { 'data': data[:] }) # making shallow copy to avoid sync and validity issues
    time.sleep(3.0) # so much hard work
    TimerQueue("update_final_results(data)", locals())

def update_interim_results(data):
    world.Note("Interim Results Available! %s" % repr(data))

def update_final_results(data):
    world.Note("Job's Done! %s" % repr(data))

def sfwlop():
    threading.Thread(target=actual_sfwlop).start()


If a shallow copy is not enough to get the job done, you can use the synchronization decorator:
from __future__ import with_statement
import functools, threading, types

def synchronized(p = None):
    def wrapping(f):
        @functools.wraps(f)
        def wrapper(*p, **kw):
            with lock:
                return f(*p, **kw)
        return wrapper
    if 'acquire' in dir(p) and 'release' in dir(p):
        lock = p
        return wrapping
    else:
        lock = threading.RLock()
    if isinstance(p, types.FunctionType):
        return wrapping(p)
    if isinstance(p, types.ClassType) or isinstance(p, object):
        for k in dir(p):
            if k not in p.__dict__:
                continue
            f = p.__dict__[k]
            if not isinstance(f, types.FunctionType):
                continue
            setattr(p, k, wrapping(f))
        return p
    return wrapping


You can use it as a function decorator:
@synchronized
def f():
    thread_name = threading.currentThread().getName()
    print "Only one thread can be in this function at any time! That thread is currently: %s" % thread_name
    time.sleep(2.0)
    print "%s leaving!" % thread_name


Or as a class decorator, the lock is shared by all of the methods:
class C:
    def f():
        thread_name = threading.currentThread().getName()
        print "Only one thread can be in this function at any time! That thread is currently: %s" % thread_name
        time.sleep(2.0)
        print "%s leaving!" % thread_name
C = synchronized(C) # have to wait for Python 2.6 for the @syntax :(


This code is fairly robust, the only problem I've seen is sometimes Bad Things Happen if you recompile the script while a child thread is still executing.
[Go to top] top

Posted by Worstje   Netherlands  (899 posts)  [Biography] bio
Date Reply #5 on Wed 12 Dec 2007 04:57 PM (UTC)

Amended on Wed 12 Dec 2007 04:59 PM (UTC) by Worstje

Message
I totally forgot about this thread, heh. I eventually found out I indeed needed threads and I got quite far, until I ran into a little issue with the background processing I wanted to do - the GIL made what I wanted to happen still freeze up my MUSHclient which kind of got me stuck. If anyone here is any good at rewriting that specific part for me in C (Isthiriel maybe? *flutter*) I'd be really grateful. I could paste all the code here, but frankly it is so damn huge and complex it would be easier explained in an IM conversation.

Any takers? ^_^
[Go to top] top

Posted by Isthiriel   (113 posts)  [Biography] bio
Date Reply #6 on Thu 13 Dec 2007 02:55 AM (UTC)

Amended on Thu 13 Dec 2007 07:21 AM (UTC) by Isthiriel

Message
My email address is @yahoo.com.au, feel free :)

EDIT: ... to email it to me.

Also, pyrex is a (mostly)python-to-c translator that can give you c-speed with (slightly non-standard) python syntax.

EDIT[2]:

Also, also, the GIL is unique to the CPython implementation. Neither IronPython nor Jython have it.

The other workaround is to start another python process and talk to it with sockets or one of the rpc systems that are part of Python's standard library.

EDIT[3]:

... which, being recognized as a Standard Solution, has been encapsulated in a module (http://www.parallelpython.com/).
[Go to top] top

Posted by Nick Gammon   Australia  (22,975 posts)  [Biography] bio   Forum Administrator
Date Reply #7 on Thu 13 Dec 2007 03:42 AM (UTC)
Message
If you are talking about a rewrite, why not use Lua coroutines? They work pretty well and there are no issues with MUSHclient locking up.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Nick Gammon   Australia  (22,975 posts)  [Biography] bio   Forum Administrator
Date Reply #8 on Thu 13 Dec 2007 03:48 AM (UTC)

Amended on Thu 13 Dec 2007 03:49 AM (UTC) by Nick Gammon

Message
Quote:

I've read up on the topic a bit and they seem like the solution I have - a lengthy process (goes beyond 5s+) that is capable of returning intermediate results. Since it is lengthy, I don't want it to block my triggers with a simple execution.


See: http://mushclient.com/forum/?id=4956

My examples there, in Lua, show how a lengthy sequence can pause for both time intervals (eg. 5 seconds) or input from the MUD (eg. waiting for some sort of result).

Also see: http://www.gammon.com.au/forum/?id=4957

That shows waiting for input from the MUD.

- Nick Gammon

www.gammon.com.au, www.mushclient.com
[Go to top] top

Posted by Worstje   Netherlands  (899 posts)  [Biography] bio
Date Reply #9 on Thu 13 Dec 2007 11:56 AM (UTC)
Message
Isthiriel:

I'll email you the relevant code in a few minutes. Thanks for wanting to look at it, I really appreciate it.

However, while I know the GIL is only a part of the CPython implementation.. isn't COM and thus a MUSHclient implementation only possible with CPython?

I have given seperate applications and such a thought, but given the fact I work a lot with references to certain objects that I'd rather not copy back and forth (we're potentially talking thousands of objects during the lengthy calculation process) since the overhead would be extreme, and performance is already a problem in its current shape.

Nick Gammon:

Heh, that is not really the purpose. It is a rewrite of the core of my curing system in one of the IRE muds, but the 700+ triggers that move the system aren't stuff I look forward to porting much. Due to the fact I have used a silly hybrid of OOP and simple procedural programming, it will be difficult enough to make all the changes I have in mind in Python. Additionally, Lua doesn't have the built-in support for set objects (no order, no duplicates) that the entire idea is founded on.
[Go to top] top

Posted by Isthiriel   (113 posts)  [Biography] bio
Date Reply #10 on Thu 13 Dec 2007 03:06 PM (UTC)
Message
Good point. It turns out neither of them can be used as WSH languages :(

Re: Lua Sets, you can roll your own with metatables.


I'm not sure what you're trying to acheive? (Possibly because I'm not familiar with the IRE affliction system.)

You're implementing a system that, when told a certain skill has been used on your character, it will learn what afflictions that skill caused by progressively attempting to cure the afflictions it knows about? (There's no way to find out what you're currently afflicted with?)

The problem with your set simplification algorithm isn't something that can be solved by reimplementing in C. The complexity is expanding combinatorially, which comes under the heading "O(NP)". C would, ideally, make it 3-4 times faster than it currently is, which means that you might be able to add an extra affliction before you reach the same slowdown.
[Go to top] top

Posted by Worstje   Netherlands  (899 posts)  [Biography] bio
Date Reply #11 on Thu 13 Dec 2007 03:35 PM (UTC)
Message
Well, being able to squeeze that bit of time out of it, and more importantly managing to keep the trigger-processing of mushclient fast and speedy while my (currently 1 processor-core) system doesn't freeze in executing triggers like my prototype tends to freeze when trying to execute new input while my background thread is running.

And yes, there is a method to find out what I need, however, that takes an equilibrium balance of half a second.. and with certain afflictions on top of it, that time can increase quite a bit. Perfecting the tracking means that I have more time to build a proper offense, especially given the fact that I am in a class who can lose a lot of headway by wasting 1-2 seconds to diagnose every few rounds.

Of course, I will diagnose when stuff gets really problematic, but in the last year there have been a lot of skills added that end up in guessing or assuming a certain affliction has happened, but having five or six twenty-choice afflictions before a horrible slow-down is a must, especially when you take in the unfortunate event of people teaming or plain group-battles.
[Go to top] top

Posted by Isthiriel   (113 posts)  [Biography] bio
Date Reply #12 on Thu 13 Dec 2007 04:16 PM (UTC)
Message
Hmm. I think I'm going to have to play one of the IRE muds before I understand the problem :(

My naive solution would be to just track one set with all of the affliction you could possibly be afflicted with and every time you attempt to cure something it is removed from the set (that is: succeed at curing or discover you don't have the affliction; attempt to cure and fail means it moves to the I-know-I-have-this-affliction set).

Or, an improvement over that would be to use a dictionary (whose keys form a set anyway) with the value for a particular affliction-key is how many skills have been used that might cause that affliction (either by incrementing for each skill that might cause it, or by adding a percentage that relates to how likely that skill is to give you that affliction -- for example injuring-a-limb-skill might have a 25% chance of afflicting each limb while blow-to-the-head skill might have a 5% chance of blindness and 95% chance of concussion; so in the first case you add 0.25 to each value for the limb keys and in the second you add 0.05 to blindness and 0.95 to concussion ... if that makes sense?), so you can then pull the keys/values as tuples and reorder them on severity and the likelihood of being afflicted and then attempt to cure them in order.

The problem that I see with the way you're trying is if skill A afflicts you with one of (l, j, k) and skill B afflicts you with one of (l, k, m) and you cure k did you remove the effects of A? or B? (or both?)

A commented game transcript would probably help me :(
[Go to top] top

Posted by Worstje   Netherlands  (899 posts)  [Biography] bio
Date Reply #13 on Thu 13 Dec 2007 05:36 PM (UTC)
Message
> Or, an improvement over that would be to use a dictionary
> (whose keys form a set anyway) with the value for a
> particular affliction-key is how many skills have been
> used that might cause that affliction (either by
> incrementing for each skill that might cause it, or by
> adding a percentage that relates to how likely that skill > is to give you that affliction .......

Problem with that is that relationships are lost between the afflictions. Some methods of curing can only be performed when you do not have other afflictions, and a 10% chance for that affliction does not properly represent the choice to make. Rspecially once you start dealing with overlapping possible afflictions (which you'll need to remove at a later stage). For example, skill A gives either x, y, or z, so you increase the chances on those, but later on, you do have a hard time determining what x and y their probability should be if said afflictions turns out to be z. Your earlier example suffers from the same dilemma: the longer you go on without a 'fresh start', the more polluted and unreliable your information becomes. (I already tried percentages to death before I arrived at this solution, heh.)

> The problem that I see with the way you're trying is if
> skill A afflicts you with one of (l, j, k) and skill B
> afflicts you with one of (l, k, m) and you cure k did you
> remove the effects of A? or B? (or both?)

Nah, I don't have that problem. For simplicity, I'll show you the output of the prototype I sent you in my email, minus some of the debugging spam.

Welcome to the affliction tester!
Try 'help' for basic instructions.
=>> import (<l, k, m>, <l, k, j>)
Interpreted set succesfully.
=>> show
Current afflictions ( 2 ):
11623696:set(['k', 'm', 'l']) (unknown range)
11623632:set(['k', 'j', 'l']) (unknown range)
=>> help identify
Forcefully identify a range into an affliction. Syntax: identify ID AFFLICTION. Or substitute ID with 'some' for the best automatic choice.
=>> identify some k
Performing identification of None into 'k'.
=>> show
Current afflictions ( 2 ):
k
11623632:set(['j', 'm', 'l']) (unknown range)
=>> discard k
Affliction 'k' removed.
This affliction stuck for 80.506 seconds.
=>> show
Current afflictions ( 1 ):
11623632:set(['j', 'm', 'l']) (unknown range)

Succesfully curing something means that you are certain you have it, thus you can identify one of the afflictions as being k. The others are adjusted so any left-over possibilities are not lost. After identification, the k option can be removed.

> A commented game transcript would probably help me :(

Well, I included the log above. A game transcript isn't really useful since I'm primarily dabbling in the higher abstractions of writing a curing system - getting uncertain afflictions, tracking them properly, being able to remove the precise affliction or all mentionings of it - without losing information that can help to better prioritize curing.

Anyhow, I hope I am not confusing you too much and I am very grateful for any (coding) insights/assistance you can offer.
[Go to top] top

Posted by David Haley   USA  (3,881 posts)  [Biography] bio
Date Reply #14 on Thu 13 Dec 2007 06:59 PM (UTC)

Amended on Thu 13 Dec 2007 07:03 PM (UTC) by David Haley

Message
Quote:
Additionally, Lua doesn't have the built-in support for set objects (no order, no duplicates) that the entire idea is founded on.

Actually, it most definitely does. :-) And no need to go into metatables, either...


myset = {}
myset[key] = true -- we just added key to the set
myset[key2] = nil -- we just removed key2 from the set

-- test for membership:
if myset[key] then
  print("the key is in the set")
end

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org
[Go to top] top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.


96,575 views.

This is page 1, subject is 3 pages long: 1 2  3  [Next page]

It is now over 60 days since the last post. This thread is closed.     [Refresh] Refresh page

Go to topic:           Search the forum


[Go to top] top

Quick links: MUSHclient. MUSHclient help. Forum shortcuts. Posting templates. Lua modules. Lua documentation.

Information and images on this site are licensed under the Creative Commons Attribution 3.0 Australia License unless stated otherwise.

[Home]


Written by Nick Gammon - 5K   profile for Nick Gammon on Stack Exchange, a network of free, community-driven Q&A sites   Marriage equality

Comments to: Gammon Software support
[RH click to get RSS URL] Forum RSS feed ( https://gammon.com.au/rss/forum.xml )

[Best viewed with any browser - 2K]    [Hosted at HostDash]