Gammon Forum : MUSHclient : General : a few questions about data flow

Log on

Gammon Forum

Entire forum

MUSHclient

General

a few questions about data flow

a few questions about data flow

It is now over 60 days since the last post. This thread is closed. Refresh page

Posted by KP (24 posts) bio

Date Tue 26 Jun 2007 08:27 PM (UTC)

Message

I've been toying with MC and perl for quite a bit and noticed a few strange things. A few notes first: my C knowledge is very rusty and I didn't look at the source yet. I think I'd figure out what exactly goes on, but it'd take me way too much time, so I'd prefer to ask here. Other than that, as I said, I'm using perl. Seeing that MC starts a new perl interpreter thread for each plugin script, I assume it will be the same for all other languages, with the only probable exception being LUA (not sure how exactly LUA scripts are parsed). This allows a bit of 'multitasking' or 'parallel processing', if you get the idea. This much said, here are my questions..

1. I've noticed that BroadcastPlugin() triggers the according function in all Plugins "simultaneously". This was confirmed by running 2 plugins and letting one of them sleep for 2 and the other for 3 seconds. Total processing time was 3 seconds. Quite logical so far, as I metioned before. But there is an interesting issue: MC seems to wait until all plugins return from the function (I see the reason for that), but if you let both send something to the output (with Note() for example), you will see that something arrives, but not what. The previous lines get "duplicated", and the output is redrawn -after- all plugins finish the job. To clarify, if you take those two plugins that sleep on BroadcastPlugin() and send back something like 'done sleeping {newline} slept X seconds' when finished, you will see something like this:


...
Old line.
 -> GO     <- some command that sends the Broadcast

...
Old line.
Old line.
Old line.  <- First plugin finished and sent back something, but its not displayed yet. Instead, the last lines get duped.

...
Old line.
PI_1 - done sleeping.
PI_1 - slept 2 seconds.
PI_2 - done sleeping. 
PI_2 - slept 3 seconds. <- all plugins done, output is rendered here.

As I said, I see the reason for waiting for all plugins to finish the job before you continue (sleep() effectively freezes MC for the time), but since MC does realize that it gets something back, why is it not rendered as soon as its received?

2. When playing with OnPluginPacketReceived(), I kind of expected all plugins to get the original packet. But instead, MC seems to 'walk' through all plugins in the order they were loaded, passing the return of the previous to the next one. This is kind of 'unreliable', because while the plugins seem to be always loaded in the order you installed them, you can mess it up by reloading a plugin (this pushes it to the end of the list). This might be screaming for a new feature like definable order in which the plugins are loaded. And I've noticed something I don't quite understand yet.. If I let both plugins report back the time when they received the packet and the time when they were done with it and were about to return it, sometimes the second plugin reports a 'received' time that is -before- the 'returned' time of the first plugin, and yet works with the modified data of the first plugin (the difference being up to 1.5 milliseconds). I can only imagine that some MC functions, like Note(), work once again in another thread and allow the language interpreter to continue before that function actually finished the work. That would be very very strange though. Any explanation to that? I can supply the testing code if any of you wish to see it.

I hope this makes any sense to you all. Thanks for your patience!

top

Posted by Nick Gammon Australia (22,982 posts) bio Forum Administrator

Date Reply #1 on Tue 26 Jun 2007 09:40 PM (UTC)

Message

Quote:

Seeing that MC starts a new perl interpreter thread for each plugin script, I assume it will be the same for all other languages, with the only probable exception being LUA (not sure how exactly LUA scripts are parsed). This allows a bit of 'multitasking' or 'parallel processing', if you get the idea. This much said, here are my questions..

All plugins have their own script state, including if you use Lua. This is in addition to the script state for the "main" world.

They are not threads. It is simply co-operative multi-tasking, if you like. At any particular moment only one will be active. If one choses to "hog" the machine (eg. by sleeping) then they all wait for it to finish.

Quote:

I've noticed that BroadcastPlugin() triggers the according function in all Plugins "simultaneously". This was confirmed by running 2 plugins and letting one of them sleep for 2 and the other for 3 seconds.

It sends the message to each plugin in sequence. As the source is publicly available you can look at it and see for yourself.


  // tell a plugin the message
  for (POSITION pluginpos = m_PluginList.GetHeadPosition(); pluginpos; )
    {
    CPlugin * pPlugin = m_PluginList.GetNext (pluginpos);


    if (!(pPlugin->m_bEnabled))   // ignore disabled plugins
      continue;

    // see what the plugin makes of this,
    pPlugin->ExecutePluginScript (ON_PLUGIN_BROADCAST,
                                  pPlugin->m_dispid_plugin_broadcast,
                                  Message, 
                                  (LPCTSTR) strCurrentID,
                                  (LPCTSTR) strCurrentName,
                                  Text); 

    if (pPlugin->m_dispid_plugin_broadcast != DISPID_UNKNOWN)
      iCount++;

    }   // end of doing each plugin

It calls the "OnPluginBroadcast" function (if it exists) in each plugin, in the order they are in the plugin list (the load order), ignoring disabled plugins. As they are not separate threads it must necessarily wait for each one to finish before moving on.

Quote:

But there is an interesting issue: MC seems to wait until all plugins return from the function (I see the reason for that), but if you let both send something to the output (with Note() for example), you will see that something arrives, but not what. The previous lines get "duplicated", and the output is redrawn

If a plugin draws to the screen (or if anything does for that matter), then it does what most Windows programs do - scrolls the output buffer the appropriate number of lines, and then "invalidates" (marks for a redraw) the parts that are new - that is, that need refreshing. This is an efficiency issue. The invalidation causes Windows to raise a "draw" event which is serviced next time through the main event loop. If every line was individually redrawn at the moment the "writing" was done, there would be much more noticeable flicker, and it would be slower. As part of the scrolling the original line is left behind, and thus you (usually for a very brief period) see a duplication.

You virtually never notice this "duplicated line" effect, it all happens so fast that the new text is drawn almost instantly. However if you do something that takes a long time, and thus delay the processing of the draw event, you can see it. For example, a long script:



for i = 1, 1000 do print ("hello") end

This will temporarily duplicate the last line on the screen, as it is scrolled up 1000 times, and then after a moment, everything is redrawn.

The design of MUSHclient (including this aspect) is what has made it consistently come out first in MUD client speed benchmarks - for drawing incoming text.

Usually with MUD text, incoming text (from the MUD) is itself a Windows event, and thus you will get something like this:

Text arrives and is processed (first event)
Line is drawn on screen
Screen scrolls up a line
Bottom line is invalidated
Main loop is re-entered
Draw event is processed (second event)
Line is drawn
Go back to point 1

Thus for normal MUD text, things are drawn "properly".

Quote:

... if you take those two plugins that sleep on BroadcastPlugin() ...

I wouldn't sleep in plugins. As I said before, that effectively hangs the client. You can make plugins effectively sleep by using Lua and making a Lua co-routine. See this thread:

http://www.gammon.com.au/forum/?id=4956

Quote:

When playing with OnPluginPacketReceived(), I kind of expected all plugins to get the original packet. But instead, MC seems to 'walk' through all plugins in the order they were loaded, passing the return of the previous to the next one.

As the documentation for OnPluginPacketReceived says:



You can return data from this function (in version 3.59 onwards). If you do, then the returned data will replace the incoming data. In other words, you can change words, omit lines, add new lines, and MUSHclient will proceed as if the altered line had arrived from the MUD.

I don't see how it could do it differently. Each plugin, since it can modify the incoming text, effectively has to change what MUSHclient considers the packet. If you happen to call OnPluginPacketReceived in multiple plugins, they each have to get the (now) modified packet.

Quote:

This is kind of 'unreliable', because while the plugins seem to be always loaded in the order you installed them, you can mess it up by reloading a plugin (this pushes it to the end of the list).

Personally I wouldn't use multiple plugins, that modify incoming packets, in a way that clashes with each other.

It would be OK if one did a "add newline to prompt", and another replaced "color" with "colour" (for example). But to make two plugins that both fiddle with incoming packets is asking for trouble IMHO.

Sure you might make a definable load order, but what do you do if two plugins both want to be first on the list?

Quote:

If I let both plugins report back the time when they received the packet and the time when they were done with it and were about to return it, sometimes the second plugin reports a 'received' time that is -before- the 'returned' time of the first plugin ...

How are you measuring the time? The loop for doing OnPluginPacketReceived is very similar to the one for OnPluginBroadcast. They each are handled in sequence. And no, Notes are not done in a separate thread.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by

Nick Gammon Australia (22,982 posts)

bio Forum Administrator

Date

Reply #2 on Tue 26 Jun 2007 10:11 PM (UTC)

Message

What you could do for plugin sequencing of packets, is make a "master" plugin that is the only one that actually handles OnPluginPacketReceived. It would probably be advisable for that plugin to "buffer" up incoming packets, so that a packet without a newline is saved until next time around, thus you are not handling half-lines.

This plugin could then consult an internal list of other plugins that are interested in modifying the packet, and do something like CallPlugin to pass the packet to that plugin (in the order in which you want to have them processed).

CallPlugin doesn't return a string, but the plugin could modify a plugin variable, that the caller could then access for the next plugin in sequence.

The list of plugins could either be hard-coded (if you are writing all of them), or done by each plugin announcing its interest (eg. at load time, or connect time, or both).

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by KP (24 posts) bio

Date Reply #3 on Wed 27 Jun 2007 02:54 AM (UTC)

Message

Thanks for you answer, Nick!

Quote:

They are not threads. It is simply co-operative multi-tasking, if you like. At any particular moment only one will be active. If one choses to "hog" the machine (eg. by sleeping) then they all wait for it to finish.

What I don't quite understand is why MC still launches them 'simultaneously'. Its easy to confirm, as I said. I can make two perlscript plugins for you if you wish (perl is almost the only language I'm good with. Been writing it for over 8 years now, and it made me lazy with everything else), but it should be reproducible with any other language too. Just use this pseudocode:


OnPluginBroadcast
  x = RANDOM(2-5)
  WAIT/SLEEP x seconds
  Note( x seconds waited )
END

You will notice that the time spent on the whole processing is not the sum of all times, but the longest time one of the plugins was idle. That is quite plausible on one hand, as the 'multi-tasking' part is not handled by MC, but happens on the windows-level. Win starts a new independant instance of the interpreter for each script (and thus each plugin), allowing them to run 'at the same time' as long as none of them eats all the CPU power. But then it would seem like MC sends the broadcast to all enabled plugins at the same time and -then- waits for them all to finish. I can't quite see that part in your snipplet, but I'll look into the whole source I guess.

Quote:

I wouldn't sleep in plugins.

That was of course not meant as a live solution. It's just that sleep is the fastest and most efficient way to simulate 'heavy work' without doing any. As I mentioned, I was only testing what and how exactly happens and how I could use it to my own advantage if possible.

Quote:

The design of MUSHclient (including this aspect) is what has made it consistently come out first in MUD client speed benchmarks - for drawing incoming text.

I see the reason of course. But what I wonder about is why MC still seems to sort the 'feedback' data in the order the plugins were launched and not in the order the data was received. Once again, let Plugin_1 sleep for 3 seconds and Plugin_2 for 1 and let both report back after they are done. You will receive a line after 1 second, but when the screen gets redrawn after another two, the message from PI_1 will be listed before the one from PI_2. And thats my main problem, not the waiting time. I'd like to see the messages in the order they are submitted if possible. (I think I failed to mention that in my first post).

Quote:

As the documentation for OnPluginPacketReceived says:
...
I don't see how it could do it differently...

Aye, sorry, I missed that part and its perfectly plausible. But the thing I'm worrying about is releasing my own plugins, that modify the packets, to the community. If someone already has a plugin that tweaks packet data, it will cause trouble. What I was suggesting is some sort of user-controlled plugin precedence, like 'up/down' buttons on the Plugins screen and/or assignable sequence numbers like those implemented for aliases and triggers. That would allow someone to 'lock' a plugin into a place and not worry about having it reloaded and messing up the whole thing. Yes, I would never let two of my own plugins modify packets, seeing that there are quite a few more efficient ways to pass around the data, but when you have multiple plugins written by different people, such problems will eventually arise.

Quote:

How are you measuring the time?

With perls Time::HiRes module, available with any basic perl installation (including ActiveState's Win32 distribution). It is very precise from what I've seen so far, and I've been using it for debugging and benchmarks for quite a while in 'real' work. I will test the whole thing again to make sure that the problem is not on my side, but it all looks fine so far. Pseudocode would be


OnPluginPacketRecieve (data)
  Note('recieved data')
  Note( time() )
  DO something WITH data
  Note('returning data')
  Note( time() )
  RETURN data
END

As you see, the 'return' time of the first plugin should never be later than 'receive' time of the second, but.. it happens. If you wish, I make a plugin that demonstrates this.

Quote:

CallPlugin doesn't return a string..

Now that would be a feature I'd kill for. Right now it is easy to 'give' data to a plugin, but its not that easy to get some back. Not without some workarounds at least. Seeing that MC waits for the call to finish anyway, why not let it return a string? It would be a great help!

Thanks again!

top

Posted by

Nick Gammon Australia (22,982 posts)

bio Forum Administrator

Date

Reply #4 on Wed 27 Jun 2007 03:53 AM (UTC)

Message

Quote:

Once again, let Plugin_1 sleep for 3 seconds and Plugin_2 for 1 and let both report back after they are done. You will receive a line after 1 second, but when the screen gets redrawn after another two, the message from PI_1 will be listed before the one from PI_2.

The plugins will be called in the list order (ie. installation order). Assuming they don't do something fancy like fire off coroutines, each plugin's function will go to completion, regardless of any delaying tactics they may use (eg. sleep).

Thus, I would expect the message from the plugin which is installed first to appear first.

Again, I would not design things that rely upon plugin installation order. If necessary make some system where messages are fed to an intermediary plugin that sorts things into the correct order.

Quote:

Yes, I would never let two of my own plugins modify packets, seeing that there are quite a few more efficient ways to pass around the data, but when you have multiple plugins written by different people, such problems will eventually arise.

True, and I don't think this can be entirely avoided. For instance, two authors may both want their plugin to be first in the list.

This is the dilemma I face in general in designing this sort of stuff. Plugins were supposed to be independent, so that this problem would not arise. Then plugin authors want to have a way of having one plugin modify a global variable, or another plugin's variables. I say "that isn't a good idea". They reply "but I WANT to".

Modifying incoming packets is obviously something that can cause Bad Things to happen. For example, you can mess up telnet sequences, MXP tags, etc.

Also, you need to be aware that packets are not "lines". A line from a MUD can span multiple packets, or one packet can contain multiple lines.

One plugin I suggested a while back tries to allow for this by batching up data from the MUD (internally in a variable) until a newline arrives, and then processing the whole thing as a line. Now it isn't going to work to install two plugins that try to do that.

Quote:

With perls Time::HiRes module, available with any basic perl installation (including ActiveState's Win32 distribution). It is very precise from what I've seen so far, ...

Try the Windows high-resolution timer, exposed through GetInfo (232). This is a very high resolution timer, that since it is built into MUSHclient, should not report different or overlapping results.

http://www.gammon.com.au/scripts/doc.php?function=GetInfo

Quote:

Seeing that MC waits for the call to finish anyway, why not let it return a string? It would be a great help!

I think I wanted to be able to return the results of the function (ie. success/failure) and a string wouldn't necessarily do that, *and* return a value (Lua can do that, but not COM, or at least, not the way I did it).

Having the plugin set one of its own variables, and the caller query that variable (GetPluginVariable) effectively lets you get as much data as you want back from a plugin. You know that after the call completes, that if the plugin was going to set the variable, it has done it by now.

http://www.gammon.com.au/scripts/doc.php?function=GetPluginVariable

Conceivably also the called plugin could itself call the caller, although I would be cautious about not setting up an infinite loop that way.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by

Nick Gammon Australia (22,982 posts)

bio Forum Administrator

Date

Reply #5 on Wed 27 Jun 2007 03:54 AM (UTC)

Message

Quote:

But then it would seem like MC sends the broadcast to all enabled plugins at the same time and -then- waits for them all to finish.

It doesn't, and I think the use of sleep is confusing things. Maybe display a dialog box, or do something that appears immediately.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by

KP (24 posts)

bio

Date

Reply #6 on Wed 27 Jun 2007 03:51 PM (UTC)

Amended on Wed 27 Jun 2007 03:56 PM (UTC) by KP

Message

Quote:

If necessary make some system where messages are fed to an intermediary plugin that sorts things into the correct order.

Yes, I guess I'll resort to that. I just wanted to avoid script 'overhead' where possible.

Quote:

For instance, two authors may both want their plugin to be first in the list.

As I mentioned, if sequence control was to be implemented, it should be controlled by the user, not the authors. You might have one plugins that tinkles with telnet/MXP and another one that just modifies the prompt for example. In such a case, a strict order would solve the problem and allow the user to have both plugins running, without any trouble. But that feature is 'might be nice to have' and far from 'necessary'.

Quote:

I say "that isn't a good idea". They reply "but I WANT to".

I think the 'problem' is that MC allows so many great things already, that any kind of limit you face feels like "Aww! But I wanna do that!" But yes, it will be better to keep a clearly defined interface that won't allow people to mess up easily. After all, you still can do very nasty and insecure things if you feel safe enough.

Quote:

Try the Windows high-resolution timer, exposed through GetInfo (232).

Indeed, I wasn't able to reproduce that behavior with it. But Time::HiRes uses the available system timer too.. Maybe its a Win32 thing. I was using Time::HiRes almost exclusively on *NIX machines and never had anything like that happen before. I'll stick to GetInfo then, thanks.

Quote:

I think I wanted to be able to return the results of the function (ie. success/failure) and a string wouldn't necessarily do that, *and* return a value (Lua can do that, but not COM, or at least, not the way I did it).

Ah true, I understand now.

Quote:

Having the plugin set one of its own variables, and the caller query that variable (GetPluginVariable) effectively lets you get as much data as you want back from a plugin. You know that after the call completes, that if the plugin was going to set the variable, it has done it by now.

Aye, and there is another 'less secure' twist to that. Using Execute() can move you into the scripting scope of another plugin, while keeping MC in the scope of the launching one. This allows you to acess the global script variables in the called plugin, while SetVariable() sets the var in callers table. Might be messy, but kind of fun to do. And its an interesting solution if you have one plugin with lots of data and a few others requesting parts of it. It's just that I wanted to avoid setting or loading MCs variables, as it is mostly temporary session data that gets passed around.

Quote:

It doesn't, and I think the use of sleep is confusing things.

Hrm.. I'll run a few more tests then. Thanks for your help!

Oh, and one more question: does BSTR have a length limit?

top

Posted by

Nick Gammon Australia (22,982 posts)

bio Forum Administrator

Date

Reply #7 on Wed 27 Jun 2007 08:43 PM (UTC)

Amended on Fri 29 Jun 2007 09:27 PM (UTC) by Nick Gammon

Message

According to the MSDN web site, the BSTR data type has a 4-byte length indicator at the start of a string, if that answers your question. :)

Internally BSTRs are generally converted to CString, which I think has a 32-bit length associated with it as well, so I think you could say "no practical limit".

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by

KP (24 posts)

bio

Date

Reply #8 on Fri 29 Jun 2007 04:03 PM (UTC)

Message

Quote:

..has a 4-byte length indicator..
..convered to CString..

Yes, that answered it fully, thanks. I was just not sure if MC uses any kind of conversion in between.

Thanks again for the speedy replies!

top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.

19,734 views.

It is now over 60 days since the last post. This thread is closed. Refresh page

Go to topic: Search the forum

top

Quick links: MUSHclient. MUSHclient help. Forum shortcuts. Posting templates. Lua modules. Lua documentation.

Information and images on this site are licensed under the Creative Commons Attribution 3.0 Australia License unless stated otherwise.