Gammon Forum : MUSHclient : Tips and tricks : PPI - A plugin communications script

Log on

Gammon Forum

Entire forum

MUSHclient

Tips and tricks

PPI - A plugin communications script

PPI - A plugin communications script

It is now over 60 days since the last post. This thread is closed. Refresh page

Pages: 1 2 3 4 5 6 7 8 9 10

Posted by

Twisol USA (2,257 posts)

bio

Date

Reply #60 on Fri 15 Jan 2010 08:04 AM (UTC)

Message

While converting my ATCP plugin to use PPI, I came across two issues: disabled service plugins (as opposed to not-installed plugins) weren't recognized in Load and caused PPI to break, and if an error occured in the service method, stuff would break in the client too. The former was rectified with an extra check in Load, and the latter with an even more minor modification in deserialize(). The updated version is linked below.

http://mushclient.pastebin.com/f16a36fbc

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

Posted by Twisol USA (2,257 posts) bio

Date Reply #61 on Sun 17 Jan 2010 01:10 AM (UTC)

Message

So what's our status here? It seemed to kind of fall off pretty quickly. I'd like to see my full-featured version included in 4.46, obviously - especially because my ATCP plugin now uses function callbacks to great effect, and I'd be loath to give that up - but I'm still very willing to change things if you think something's out of place.

I also added a small bit to _G[request_msg] to pass the calling plugin's ID as the first parameter, because I can see very real instances when you'd want to know the caller's ID.

  -- Call method, get return values
  local returns = {func(id, unpack(params))}
  
  -- Send returns
  send_params(returns)
end

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

Posted by Twisol USA (2,257 posts) bio

Date Reply #62 on Sun 17 Jan 2010 09:54 AM (UTC)

Message

As a side-note, I remembered us discussing module() earlier in the thread, and I came up with this function to create a new ready-to-use environment containing only what would have come with the environment to begin with. It's not perfect, of course - it shares the same subtables with the original _G. But it does do a good job of creating a standalone environment that a library can use without worrying about locals.

function basefenv()
  local _G = getfenv(1)
  
  local _G2 = {
    __VERSION = _G.__VERSION,
    assert = _G.assert,
    collectgarbage = _G.collectgarbage,
    dofile = _G.dofile,
    error = _G.error,
    getfenv = _G.getfenv,
    getmetatable = _G.getmetatable,
    ipairs = _G.ipairs,
    load = _G.load,
    loadfile = _G.loadfile,
    loadstring = _G.loadstring,
    module = _G.module,
    next = _G.next,
    pairs = _G.pairs,
    pcall = _G.pcall,
    print = _G.print,
    rawequal = _G.rawequal,
    rawget = _G.rawget,
    rawset = _G.rawset,
    require = _G.require,
    select = _G.select,
    setfenv = _G.setfenv,
    setmetatable = _G.setmetatable,
    tonumber = _G.tonumber,
    tostring = _G.tostring,
    type = _G.type,
    unpack = _G.unpack,
    xpcall = _G.xpcall,
    
    bc = _G.bc,
    bit = _G.bit,
    coroutine = _G.coroutine,
    debug = _G.debug,
    io = _G.io,
    lpeg = _G.lpeg,
    math = _G.math,
    os = _G.os,
    package = _G.package,
    progress = _G.progress,
    rex = _G.rex,
    sqlite3 = _G.sqlite3,
    string = _G.string,
    table = _G.table,
    utils = _G.utils,
    world = _G.world,
    
    check = _G.check,
    gcinfo = _G.gcinfo,
    loadlib = _G.loadlib,
    
    alias_flag = _G.alias_flag,
    colour_names = _G.colour_names,
    custom_colour = _G.custom_colour,
    error_code = _G.error_code,
    error_desc = _G.error_desc,
    extended_colours = _G.extended_colours,
    sendto = _G.sendto,
    timer_flag = _G.timer_flag,
    trigger_flag = _G.trigger_flag
  }
  _G2._G = _G2
  
  return setmetatable(_G2, getmetatable(_G))
end

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

Posted by

Nick Gammon Australia (22,975 posts)

bio Forum Administrator

Date

Reply #63 on Mon 18 Jan 2010 03:02 AM (UTC)

Message

Twisol said:

So what's our status here? It seemed to kind of fall off pretty quickly.

Well I was waiting to see what happened, and if anyone else would comment.

My attitude towards incorporating verbatim your code in the next release can be summed up somewhat by the comments made by the Lua developers about accepting patches, on this page:

http://www.lua.org/faq.html#1.9

Lua developers said:

1.9 Do you accept patches?

We encourage discussions based on tested code solutions for problems and enhancements, but we never incorporate third-party code verbatim. We always try to understand the issue and the proposed solution and then, if we choose to address the issue, we provide our own code.

I am very grateful to you for raising the issue of plugin-to-plugin communication. However I believe that my simplified version addresses the situation adequately. So far, out of the thousands of MUSHclient users, and the dozens of plugin developers who frequent these forum pages, there has been almost complete silence about whether this module addresses any needs that are currently considered requiring addressing.

Since I believe my version provides a subset of your version, an upgrade could be considered in future if it was needed (although your latest patch which supplied the ID of the calling module might break this).

As a general rule, if there are two solutions to a problem, I prefer the simpler one. In years to come the simpler one is generally easier to understand, and to fix or modify if needed.

In any case, as this module is not part of the core MUSHclient code (that is, the .exe file), if you required your more sophisticated version when distributing plugins, you can, of course, include it.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by

Twisol USA (2,257 posts)

bio

Date

Reply #64 on Mon 18 Jan 2010 05:10 AM (UTC)

Message

To me, it's not about seeing a need and filling only the exact requirements of it, it's about providing options that might spark more ideas. Anyways, if I had known this was about getting voices, I'd have pointed this thread out to the plugin developers in Achaea who don't frequent these forums by any stretch of the imagination.

I would like to ask what you mean by "simple", though. Simple to the end user, or in the implementation?

I agree that I could still distribute my own version, but that would still require an extra download for the users of the plugins (not just the developers), which was the driving force behind my creating LoadPPI originally. Both of our versions can be used exactly the same way (minus the ID patch), with exactly the same semantics, except that mine is simply more flexible. I'm willing to take the hit in code 'complexity' (and my version isn't very complex, IMO) if it means the user has an easier time of it.

I know I'm beating a dead horse, I'm sorry about that. It's not really about the code, I just honestly don't understand... You're leaving out features that I think are really useful, and I've used them in a plugin (ATCP) that benefits greatly from said features (mainly function callbacks as parameters), for no other reason than simplicity in the module itself? Again, I'm not trying to flame you, troll you, or be a bad sport in general. I just don't understand, and that's what I'm trying to figure out.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

Posted by Nick Gammon Australia (22,975 posts) bio Forum Administrator

Date

Reply #65 on Mon 18 Jan 2010 08:40 PM (UTC)

Amended on Mon 18 Jan 2010 09:24 PM (UTC) by Nick Gammon

Message

OK, well it would help if you documented what your version does exactly, and what the extra features are. A programming exercise is not really done until it is documented, otherwise people don't know what the code does, how to use it, and why they might want to.

My proposed version on page 2 of this thread has about 80 lines of documentation at the start (as a comment), yours has none.

In the middle, for example, your module has "Called to access and return a value from the service". What does this do? Is this on top of calling the function itself, but gets a value separately? Why do this? Can you explain in some detail all these extra features and what they are useful for?

Given that both yours and my module achieve approximately the same thing (simplify calling a function in another plugin, passing multiple arguments, and getting multiple values as a return), I note that my version, excluding the initial comments, is 90 lines of code whereas yours is 374 lines.
My tests showed my version worked about 4 times as fast as yours.
My version only uses a single MUSHclient variable for each call (for all the return values), whereas yours uses a variable per argument, and another variable for each return value.
You have rewritten serialization for reasons that are a bit obscure to me, considering there is already a "serialize" module. The existing serialize module has had the test of time, in terms of ironing out wrinkles.
You say you want to be able to serialize tables with self-references, or references to other tables, but neither David Haley nor I can see why you would want to do that, apart from showing it can be done. In any case, the existing serialize module will do that if you change from serialize.save_simple to just serialize.save.
Can some of the extra features, like finding what a plugin supports, not be layered on top of my simpler implementation, if required? And would they be required?
I find code like this just obscure:
```
 PPI_list[id] = tbl
 PPI_list[tbl.ppi] = id
```
It appears to be storing A inside B, and then B inside A. I'm not sure why this is required.
You mention an important feature is "function callbacks as parameters". Can you explain that? Give an example? Are you passing a function from one plugin to another? Is this required?

I would like a bit more peer review of this code (including mine). In other words, an assessment by other plugin-writers, than you or me.

Every couple of days you have been posting bug fixes or minor improvements. One thing at least I would like to see if a week or two elapse where no further changes were made (obviously you could achieve this by simply not posting) but not only that, to have other people use it and say "yes it works as advertised". Of course, to work as advertised you need to do what I said at the start of this post, and document exactly what it does, giving examples of usage.

- Nick Gammon

www.gammon.com.au, www.mushclient.com

top

Posted by

Twisol USA (2,257 posts)

bio

Date

Reply #66 on Mon 18 Jan 2010 09:14 PM (UTC)

Message

Alright, this is more like it. I'll get back to you ASAP once I've finished. ;)

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

Posted by Twisol USA (2,257 posts) bio

Date

Reply #67 on Mon 18 Jan 2010 10:43 PM (UTC)

Amended on Mon 18 Jan 2010 10:45 PM (UTC) by Twisol

Message

Firstly, here's an in-depth look at the module itself. I hope it helps explain things for you. I'll go back over each point you raised in a separate post.

My PPI implementation communicates with other PPIs by means of three 'messages': ACCESS, REQUEST, and CLEANUP. REQUEST should arguably be INVOKE, but it's a remnant of the original implementation. These messages are sent through CallPlugin(), passing the client plugin ID as its single parameter.

ACCESS: Retreives a value from the service by name. The value can be any of: string, number, boolean, table, function.
REQUEST: Invokes a remote function by ID. Parameters are serialized into MUSHclient variables for access by the service. Return values are serialized by the service and retreived afterwards. Any of the following values can be passed or returned: string, number, boolean, table, function.
CLEANUP: Called after every ACCESS or REQUEST message to remove the temporary variables involved.

All variables are passed by value. Modifications to a local copy have no effect on the original value.

The serialization format for an individual value is simple enough:

Strings: "s:a string"
Numbers: "n:42"
Booleans: "b:1"
Tables: "t:1" *
Functions: "f:1" **
Nil, or unknown: "z:~"

* Tables are serialized separately, and each is given its own MUSHclient variable and an ID. It is very important to note all subtables within a table are serialized as well, so serializing _G would probably be a bad idea.

** Functions are not serialized, per se; they are stored locally and given a unique identifier, which is used in the serialized list of parameters. When deserialized, a new resolver function is created, which when invoked will send a REQUEST message on that method ID to the service.

These serialized values are inserted into a MUSHclient array and exported, resulting in a string like this:

n:1|s:a string|n:2|b:1

The keys are also serialized, as it is legal to serialize a table with either string or numeric keys. The above line could be seen more as:

n:1 | s:a string

n:2 | b:1

In order to communicate with another plugin in the first place, you must create a PPI proxy. This is done with PPI.Load(), using the ID of the service plugin as the only parameter:

test = PPI.Load("7f88d638ba84f0c48b646dd2")

A simple access and subsequent invocation looks like this:

local tbl_param = {4}
test.FooBar(true, "2", 3, tbl_param, tbl_param)

Breaking it down, we have this series of steps:

1. "test.FooBar" causes an ACCESS message to fire. It retrieves the value stored remotely as FooBar, which could be any of the acceptable types mentioned previously. In this case, we assume it's a function.

2. "FooBar(...)" causes a REQUEST message to fire through the retrieved method. The arguments are serialized together as a table. The resulting variables used would look like this:

PPIparams1: n:1|f:1|n:2|b:1|n:3|s:2|n:4|n:3|n:5|t:2|n:6|t:2

PPIparams2: n:1|n:4

As you can see, the entire original parameters list itself is serialized as a single variable. Extra variables are used as needed to serialize nested tables, such as tbl_param. As well, even if a subtable appears more than once, it is only ever serialized once.

The function ID being invoked is passed as the first parameter, which is why there are six items rather than five.

A service exposes values to the world by using PPI.Expose(). This method takes two parameters: the key by which clients will retreive your value, and the value to store.

_G.callbacks_list = {}

PPI.Expose("number", 42)
PPI.Expose("boolean", true)
PPI.Expose("string", "My Plugin")
PPI.Expose("table", {
  ["a key"] = "a value",
  [1] = 10,
  [2] = 50,
})
PPI.Expose("StoreCallback", 
  function(id, callback)
    table.insert(_G.callbacks_list, callback)
  end
)
PPI.Expose("ExecuteCallbacks",
  function(id, ...)
    for _,v in ipairs(_G.callbacks_list) do
      v(...)
    end
  end
)

This code exposes a series of values, covering every type supported by PPI. Here is an example client and its output.

test = PPI.Load("7f88d638ba84f0c48b646dd2")
require('tprint')

print("boolean: ", test.boolean)
print("string: ", test.string)
print("table:")
tprint(test.table)
print()
print("function: ", test.StoreCallback)

myfoo = 10

test.StoreCallback(function(id, ...) tprint{...} end)
test.StoreCallback(function(id, ...) print() print(myfoo, " - ", #{...}) end)
test.ExecuteCallbacks(10, 15, 30, "woohoo")

Output:

function:  function: 01459910

boolean:  true

string:  My Plugin

table:

1=10

2=50

"a key"="a value"



function:  function: 014598E0

1=10

2=15

3=30

4="woohoo"



10  -  4

And of course, if you try to access a value that's nonexistant, the client gets nil. The passing of functions (or more accurately function proxies) allows plugin developers to make use of the subscription model. Just like BroadcastPlugin() sends data to all plugins (interested or not), a service can expose a 'subscription' method for plugins to pass a callback to, and that plugin can later execute those callbacks it received.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

Posted by Twisol USA (2,257 posts) bio

Date

Reply #68 on Tue 19 Jan 2010 12:21 AM (UTC)

Amended on Tue 19 Jan 2010 05:00 AM (UTC) by Twisol

Message

I don't mind putting some tutorial documentation in the module, but I prefer to keep it separate from the file, such as online, for more comfortable reading.

Nick Gammon said:
Given that both yours and my module achieve approximately the same thing (simplify calling a function in another plugin, passing multiple arguments, and getting multiple values as a return), I note that my version, excluding the initial comments, is 90 lines of code whereas yours is 374 lines.

I personally don't think code size is a good comparison. Mine has more lines - many of which are whitespace and comments, by the way - mainly because I wrote my own serialization functions (in order to 'serialize' functions properly). A good design also tends to be 'larger' in general, but it's also easier to extend/maintain. (I'd like to think this is designed at least decently.)

Nick Gammon said:
My tests showed my version worked about 4 times as fast as yours.

Correction: maybe twice as fast when compared to the version caching the retrieved method. Your version sends only one CallPlugin message, with simpler serialization. Mine sends two (one access, one request/invoke), with slightly more complex serialization to accomodate PPI-specific requirements. In my first test, both access and request messages were sent every iteration. In the second, I cached the access message first in a local, then used that in the loop. That's standard practice in Lua anyways (we discussed this in relation to module() and storing methods in locals before calling it).

Nick Gammon said:
My version only uses a single MUSHclient variable for each call (for all the return values), whereas yours uses a variable per argument, and another variable for each return value.

Incorrect. Mine uses a variable per table. The parameters list is treated as a single table itself, so if you pass no tables as parameters, no extra variables are used. This applies identically to the return values. Yes, in a past version I used a variable for each argument/return, but that's no longer the case.

Nick Gammon said:
You have rewritten serialization for reasons that are a bit obscure to me, considering there is already a "serialize" module. The existing serialize module has had the test of time, in terms of ironing out wrinkles.

True. However, I wanted the PPI communications protocol to be as unhindered by language as possible, and I also wanted to be able to allow functions (or rather, function identifiers) to be passed. It turned out to be easiest to write my own, but if you see any flaws or bottlenecks in my implementation, please let me know.

Nick Gammon said:
You say you want to be able to serialize tables with self-references, or references to other tables, but neither David Haley nor I can see why you would want to do that, apart from showing it can be done. In any case, the existing serialize module will do that if you change from serialize.save_simple to just serialize.save.

In reference to the bolded... excuse me if I sound a bit tweaked, but there is no difference between a table containing a table, and a table containing a reference to another table. Clearly, we want to be able to pass tables containing tables, you have said as much yourself. I'm not understanding this.

In relation to the rest of the quote, yes, serialize.save will do the job properly and without subtable duplication, but I found it was easiest to roll my own when it came to the PPI communication format.

Nick Gammon said:
Can some of the extra features, like finding what a plugin supports, not be layered on top of my simpler implementation, if required? And would they be required?

I'm not sure what you mean by "finding what a plugin supports". The old SUPPORTS from before was adapted into the current ACCESS, which simply retrieves a value from the PPI, whether it be typical values or a method. Any kind of 'supports' checking is simply done by checking the value against nil.

Nick Gammon said:
*I find code like this just obscure:


 PPI_list[id] = tbl
 PPI_list[tbl.ppi] = id

It appears to be storing A inside B, and then B inside A. I'm not sure why this is required.

Incorrect, although I agree that it isn't clear. I'm storing two sets of data in the PPI_list table: one, a list of client PPI records (including private data) keyed by service ID; and two, the ID of the client PPI keyed by the actual PPI object contained within the record (the one that the user is given). It's confusing, but the items in the record ('tbl') are private data, and tbl.ppi is the PPI proxy the user uses. For the most part, the second item (the id keyed by the PPI) is used in the PPI metatable's __index in order to get at the private data. I could have used a separate table to store the ppi-to-id mappings, but it was easier just to do this, since the keys couldn't ever clash.

Your analogy would be more A.c = B; B.c = A. That's not the case, because here it's the same table being inserted into both times. It just happens to be odd keying.

Nick Gammon said:
*You mention an important feature is "function callbacks as parameters". Can you explain that? Give an example? Are you passing a function from one plugin to another? Is this required?

I think my above overview showed a good example of this. Yes, it's absolutely required in order to sanely implement selective broadcasting (aka event subscription). My ATCP plugin uses it: a client plugin just tells it what message it wants to listen for, and a callback to call when it comes in. It's very clean and clear.

EDIT: Rather belated fix from 'model' to 'module'.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

Posted by

David Haley USA (3,881 posts)

bio

Date

Reply #69 on Tue 19 Jan 2010 05:13 AM (UTC)

Message

Quote:
ACCESS: Retreives a value from the service by name. The value can be any of: string, number, boolean, table, function.
REQUEST: Invokes a remote function by ID. Parameters are serialized into MUSHclient variables for access by the service. Return values are serialized by the service and retreived afterwards. Any of the following values can be passed or returned: string, number, boolean, table, function.
CLEANUP: Called after every ACCESS or REQUEST message to remove the temporary variables involved.

An internal 'cleanup' might or might not be needed for implementation reasons, but these are precisely implementation details and the user should never, ever have to worry about cleaning up implementation-specific temporaries.

As soon as I 'access' data or 'request' a function's return value, I have the values in hand and they should go away immediately in any other form they might exist in other than the values I have.

Quote:
with slightly more complex serialization to accomodate PPI-specific requirements

It's still a little unclear to me, to be honest, if these requirements were driven by actual problems to solve, or problems that could be solved.

Quote:
but there is no difference between a table containing a table, and a table containing a reference to another table.

Actually, there sort of is. You can think of it as the difference between a containment tree and a containment graph. I think we all agree that we want to be able to pass tables with subtables; it's still unclear to me (and Nick, I believe) why we want tables that can refer to themselves (which is what arbitrary table references allow).

Quote:
but I found it was easiest to roll my own

A general rule of thumb is to avoid rolling your own unless you really, really have to.

For whatever it's worth, your serialization code (as posted on page 2, v1.0.1, was the most recent I saw in the thread) does not allow a table key to be anything but a string or number, whereas Nick's can (with some limitations). Before you point at the limitations, I would continue to argue that limitations are only a problem insofar as they prevent the tasks we actually have from being accomplished. To be fair, table keys of type 'table' are perhaps weird in the first place, but part of your argument is being language-independent and as general as possible. So this could be a source of confusion. In fact, your serialization function silently ignores stuff it can't serialize, which could lead to very confusing problems!

TBH, and I hate to harp on this, but it still seems to me like the requirements haven't really been defined. I'm not talking about implementation requirements, I'm talking about API requirements. We're talking of "PPI requirements" but I haven't seen a single place where the high-level goals are cleanly laid out, with real-world use cases and justifications for each of the requirements.

Complexity in general, solving problems simply because you can, is not always a good thing. This is a Very Important Concept and perhaps hard to explain succinctly. There's a good quote about perfection by Antoine de Saint-Exupery; paraphrasing, the designer knows that he has achieved perfection not when there is nothing left to add but when there is nothing left to take away. This is extraordinarily relevant to computer science in general and API/protocol/etc. design in particular. Solving all kinds of problems sounds awfully nice, until you try to solve so many that you end up confusing the user when it comes to solving the problems you initially had. This is a somewhat lengthy reference, but I would recommend this document:
http://cacm.acm.org/magazines/2009/5/24646-api-design-matters/fulltext
(Caveat: I've only read about half of it so far, myself)

To summarize: I find it very difficult to comment without having a very clear picture of the exact problem we're solving and why it's a problem. This should be general, but more specific than just "sending stuff between plugins" -- that much is obvious enough. ;-)

Here's an example: we need to send functions around. Do we really need to send functions that don't have proper names? If we only send properly named functions, then we can dramatically simplify the passing-around of functions; namely, you specify them by name and you're done. Yes, yes, it would be nice to construct arbitrary callbacks in Lua using closures etc. and then be able to pass these to VBscript or whatever. But this introduces complexity for the entire library, for a very specific use case -- one that could, incidentally, be solved by the user by giving names to these lambdas in the first place. Yes, this punishes the user who has this (IMHO rather funky requirement), but the alternative is to punish everybody by making a generally more complex system.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org

top

Posted by

Twisol USA (2,257 posts)

bio

Date

Reply #70 on Tue 19 Jan 2010 05:35 AM (UTC)

Amended on Tue 19 Jan 2010 05:40 AM (UTC) by Twisol

Message

David Haley said:
An internal 'cleanup' might or might not be needed for implementation reasons, but these are precisely implementation details and the user should never, ever have to worry about cleaning up implementation-specific temporaries.

As soon as I 'access' data or 'request' a function's return value, I have the values in hand and they should go away immediately in any other form they might exist in other than the values I have.

You misunderstand me, these ARE implementation details. The user only has to deal with PPI.Load, PPI.Expose, and whatever methods the service exposes. I explained them because it's critical to understand how PPI actually works, in the context of this discussion.

David Haley said:

Quote:
with slightly more complex serialization to accomodate PPI-specific requirements

It's still a little unclear to me, to be honest, if these requirements were driven by actual problems to solve, or problems that could be solved.

I mentioned function callbacks, which have a very clear use that I've mentioned and utilized already. If there's a specific other requirement I mentioned that you don't understand the need for, I'd be glad to address it directly.

David Haley said:

Quote:
but there is no difference between a table containing a table, and a table containing a reference to another table.

I'm still not understanding, sorry... If you have a table, and you serialize subtables (something we agree we need), we don't want to serialize the same subtable multiple times if it appears more than once (i.e. a graph rather than a tree, as you said). This requires a cache mechanism, which I use. At this point, tables that refer to themselves - or simply cyclic references in general - are valid simply because of the cache. I certainly wasn't thinking of self-referential tables when I wrote the code.

David Haley said:

Quote:
but I found it was easiest to roll my own

I hate to say it, but you're several versions behind. I began pasting them elsewhere and linking to them after they got too large for the posts. The latest version is here:

http://mushclient.pastebin.com/f16a36fbc

In relation to things that can't be serialized, my version uses z:~ in its place, which is just a serialized nil.

David Haley said:
TBH, and I hate to harp on this, but it still seems to me like the requirements haven't really been defined. I'm not talking about implementation requirements, I'm talking about API requirements. We're talking of "PPI requirements" but I haven't seen a single place where the high-level goals are cleanly laid out, with real-world use cases and justifications for each of the requirements.

I think in a very, um... different manner than most people I've met. It's very hard for me to lay things out so concretely. I tend to work by examples, such as my ATCP plugin - I posted the new version a day or two ago in its own thread. Most examples are also in my head as I work (and I do throw out ideas that simply aren't useful). If you have any concrete questions about the API, rather than asking me to lay it out for you, I would be happy to oblige. >_>

David Haley said:
Complexity in general, solving problems simply because you can, is not always a good thing. This is a Very Important Concept and perhaps hard to explain succinctly. There's a good quote about perfection by Antoine de Saint-Exupery; paraphrasing, the designer knows that he has achieved perfection not when there is nothing left to add but when there is nothing left to take away. This is extraordinarily relevant to computer science in general and API/protocol/etc. design in particular. Solving all kinds of problems sounds awfully nice, until you try to solve so many that you end up confusing the user when it comes to solving the problems you initially had. This is a somewhat lengthy reference, but I would recommend this document:
http://cacm.acm.org/magazines/2009/5/24646-api-design-matters/fulltext
(Caveat: I've only read about half of it so far, myself)

When I look at PPI, I believe that everything has a very important use case. It all works through the same code system. If I had to, I could limit PPI.Expose to methods again, but I don't see the point: it all works through the same core system. PPI.Expose is like passing a parameter; using a client PPI to access it is like getting the parameter from a function call. It doesn't make sense, to me, to limit it in such a way. It would be adding to the code's burden, rather than removing from it.

David Haley said:
To summarize: I find it very difficult to comment without having a very clear picture of the exact problem we're solving and why it's a problem. This should be general, but more specific than just "sending stuff between plugins" -- that much is obvious enough. ;-)

Like I said earlier, it's kind of hard to speak concretely about it unless I have an applicable question. >_> I know what I expect of PPI, but it's hard for me to communicate it.

David Haley said:
Here's an example: we need to send functions around. Do we really need to send functions that don't have proper names? If we only send properly named functions, then we can dramatically simplify the passing-around of functions; namely, you specify them by name and you're done.

Sorry, what? It doesn't matter if it has a name or not, you're giving the function value itself to PPI. And it does give it a name, or more properly, an identifier. The identifier could have been a string, but numbers lent themselves better to uniqueness: just increment the last one. Strings just didn't seem to fit anyways.

David Haley said:
Yes, yes, it would be nice to construct arbitrary callbacks in Lua using closures etc. and then be able to pass these to VBscript or whatever. But this introduces complexity for the entire library, for a very specific use case -- one that could, incidentally, be solved by the user by giving names to these lambdas in the first place. Yes, this punishes the user who has this (IMHO rather funky requirement), but the alternative is to punish everybody by making a generally more complex system.

Functions are not actually passed between plugins. Identifiers are. Any language could theoretically save this identifier somewhere, and when the user is ready, send that identifier back across with a REQUEST message.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

Posted by David Haley USA (3,881 posts) bio

Date Reply #71 on Tue 19 Jan 2010 03:39 PM (UTC)

Message

Quote:
I explained them because it's critical to understand how PPI actually works, in the context of this discussion.

This is the kind of thing that should be made explicit, probably, as you describe things. You have two audiences: (1) people who develop PPI (which is really you, and Nick who will eventually approve something for inclusion into MUSHclient) and (2) people who use it.
You need to motivate all of this for both groups, and different motivations are needed for each.

Quote:
I mentioned function callbacks, which have a very clear use that I've mentioned and utilized already.

These can be achieved without rolling yet another serialization library (see below).

Quote:
If you have a table, and you serialize subtables (something we agree we need), we don't want to serialize the same subtable multiple times if it appears more than once (i.e. a graph rather than a tree, as you said).

If you only have a tree-table structure, you will never have repeated tables. This is what Nick means when he talks about tables containing subtables (tree structure) and self-referential structures (graph structure). (Technically, you can have a graph without self-referential tables; all you need is a repeated table.) What Nick and I are saying is that we don't really see the need for repeated tables.

Quote:
I hate to say it, but you're several versions behind.

The version you linked to does not allow table keys, either. In particular,


    local key = nil
    if type(k) == "string" then
      key = "s:" .. k
    elseif type(k) == "number" then
      key = "n:" .. k
    end

It's pretty apparent that if type(k) is anything other than string or number, 'key' remains nil; looking at the lines right below that, ArraySet is only called when key is not nil. And, it's all silently ignored.

Quote:
If you have any concrete questions about the API, rather than asking me to lay it out for you, I would be happy to oblige.

Sigh. :(
Well, this is complicated then. :) You're trying to convince people that your solution is inherently superior, despite their reservations that it's too complex for the problem at hand. There is really no other way for you to convince this than to "lay it out". :-/

The point is that the questions aren't about the API's details, it's about whether this is the appropriate solution in the first place.

Quote:
Sorry, what? It doesn't matter if it has a name or not, you're giving the function value itself to PPI. And it does give it a name, or more properly, an identifier. The identifier could have been a string, but numbers lent themselves better to uniqueness: just increment the last one. Strings just didn't seem to fit anyways.

This is the kind of stuff I'm talking about... there's really no need to send the function value. It's solving a problem just because you can, not because it's a problem that needs to be solved. It's overly complicated and only helpful in a very, very small number of cases. In the vast majority of cases, the function's name is sufficient, and it considerably simplifies things.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org

top

Posted by Twisol USA (2,257 posts) bio

Date

Reply #72 on Tue 19 Jan 2010 05:26 PM (UTC)

Amended on Tue 19 Jan 2010 05:41 PM (UTC) by Twisol

Message

David Haley said:

This is the kind of stuff I'm talking about... there's really no need to send the function value. It's solving a problem just because you can, not because it's a problem that needs to be solved. It's overly complicated and only helpful in a very, very small number of cases. In the vast majority of cases, the function's name is sufficient, and it considerably simplifies things.

I'll respond to the rest later, because I'm tired atm, but I'm NOT sending its value by any stretch of the imagination. I am storing its value locally within the plugin that's sending it, and sending an identifier that stands in for the function. For all intents and purposes, that is a name. It's not the user's direct name for it, but it is absolutely a name.

EDIT after shower: Serializing a function's actual value would be catastrophic on the function's actual use as a communication device. It would keelhaul upvalues, for one. No, if I serialize its actual value, my above examples wouldn't work, as I explicitly placed an extra variable just for one of the callbacks to print out.

For why I store the function before sending an identifier, it's like this:

a = function() end
b = a
a = nil

'b' is unaffected, no? It still has a valid reference to the function. If I referred directly to the variable the user sent, then by modifying or removing that variable, the plugin it was sent to suddenly has a bad reference. Simply put, I am trying to preserve natural Lua semantics.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

Posted by Twisol USA (2,257 posts) bio

Date

Reply #73 on Tue 19 Jan 2010 05:54 PM (UTC)

Amended on Tue 19 Jan 2010 06:13 PM (UTC) by Twisol

Message

David Haley said:

Quote:
I explained them because it's critical to understand how PPI actually works, in the context of this discussion.

My apologies, then - I didn't think I was supposed to be catering to users at this point. I was just trying to explain my version to Nick as per his questions.

David Haley said:

Quote:
I mentioned function callbacks, which have a very clear use that I've mentioned and utilized already.

These can be achieved without rolling yet another serialization library (see below).

I need somewhere to put the function and replace it with an identifier, and I have a language-agnostic serialization protocol that needs to be supported. The serialize module wouldn't cut it because of the latter, and I also specifically wanted to limit the types of keys allowed.

David Haley said:

My goal is to retain normal Lua semantics as much as possible. It was a very simple addition, and I think it is a very useful addition. Furthermore, I needed the cache in order to separate tables out so that they all go in separate variables. You could argue that you could put them all in one variable, but (1) I don't like at all the sound of that approach, it's too hacked; and (2) as more tables are added, more and more backslashes are added by ArrayExport() to escape the | separator in the arrays. Wimpy arguments? Maybe. But there's no gain to doing it that way, either.

David Haley said:

Quote:
I hate to say it, but you're several versions behind.

The version you linked to does not allow table keys, either. In particular,


    local key = nil
    if type(k) == "string" then
      key = "s:" .. k
    elseif type(k) == "number" then
      key = "n:" .. k
    end

Tables as keys is one of those things that I thought to myself and said, no, that's not going to be useful at all in this context. First and foremost, very, very few languages can use an arbitrary type as a key. Secondly, table keys are even more unlikely to be supported. Thirdly, even if I did allow table keys, all passed tables are copies. You would have to run over pairs() in the first place to be able to get at them, which defeats the purpose of tabular keys.

David Haley said:

Quote:
If you have any concrete questions about the API, rather than asking me to lay it out for you, I would be happy to oblige.

I would honestly like to ask you what your own solution would be. I get the feeling that maybe I wrote PPI too 'cleverly', which can be a very bad thing, and if you have any ideas I would like to hear them. I'm trying to explain PPI by answering your and Nick's criticisms and questions, because it comes easily to me when responding.

'Soludra' on Achaea

Blog: http://jonathan.com/
GitHub: http://github.com/Twisol

top

Posted by

David Haley USA (3,881 posts)

bio

Date

Reply #74 on Tue 19 Jan 2010 06:27 PM (UTC)

Message

Quote:
but I'm NOT sending its value by any stretch of the imagination. I am storing its value locally within the plugin that's sending it, and sending an identifier that stands in for the function.

I didn't say you were; I'm saying that this extra indirection is unnecessarily complicated. I do not believe that it is a common use case to want to send a function value rather than the (user-space) name of a function.

For example, you later say: "I need somewhere to put the function and replace it with an identifier".
But this complexity wouldn't exist if we (as users) were only sending function names, as opposed to function values. (That the function values are internally converted to random identifiers is irrelevant at present. As far as the user is concerned, values are being sent.)

Quote:
Simply put, I am trying to preserve natural Lua semantics.

Then this is confusing to me w.r.t. making a language agnostic cross-plugin communication layer.

Is the problem to deal with cross-plugin, cross-language communication, or is the problem to deal with Lua communication?

Quote:
My apologies, then - I didn't think I was supposed to be catering to users at this point.

You are necessarily dealing with (even hypothetical) users as soon as you start talking about designing an API to be included in the MUSHclient "core". An API without users is not a useful project. The argument being made against the existing API is that it seems to complex for what users are likely to need; the response (from you) must argue that users in fact will need this complexity often enough for it to become a requirement.

Quote:
Wimpy arguments? Maybe. But there's no gain to doing it that way, either.

I'm not sure which way you're referring to with "that way", or if you're responding to me in general; I was talking about the unclear value of allowing graph-like tables as opposed to tree-like tables, which simplifies implementation.

Quote:
First and foremost, very, very few languages can use an arbitrary type as a key.

Not really. C++, Java, Perl, Lua, Python, Ruby, to name just a few, all allow tables as keys (with restrictions in some cases).

All I'm trying to say is that it seems somewhat arbitrary to argue for generality in one case and then to get rid of another generality by saying it's specific.

I mean, if we're worried about serializing complex tables with cyclical structures and all that, and especially if we're worrying about maintaining Lua semantics, then in fact it seems very reasonable to want to keep Lua semantics in this case. Otherwise, the API is behaving inconsistently: it's keeping the semantics here, but dropping them there -- and silently dropping them, too!

Quote:
I would honestly like to ask you what your own solution would be.

My honest response is that I don't really know because I do not feel that there has been an adequate presentation of what users of cross-plugin communication actually need. For example, I don't know if a low-level communication layer is what we want, or if we want a higher-level semantics transport layer. My first and strongest gut reaction is to avoid complexity unless it is clear that it is unavoidable. My second gut reaction is to separate the implementation from the concept. There's been a lot of talk about implementation so far, but relatively little talk about the problems we're trying to solve. Examples:
Why do we want to send complex tables?
Why do we want to send function values (remember, from the user's perspective), and not function names?
What are we trying to send in the first place?
Why do we need plugins to talk to each other?

For technical questions, "because we can" is not an acceptable answer unless it adds no complexity to the end result.

I find it remarkably difficult, if not basically impossible, to design a system when you don't know exactly what you're designing to. What are the user's requirements? These are different from an implementation's requirements, which should be driven by the user's requirements in the first place.

David Haley aka Ksilyan
Head Programmer,
Legends of the Darkstone

http://david.the-haleys.org

top

The dates and times for posts above are shown in Universal Co-ordinated Time (UTC).

To show them in your local time you can join the forum, and then set the 'time correction' field in your profile to the number of hours difference between your location and UTC time.

276,850 views.

This is page 5, subject is 10 pages long: 1 2 3 4 5 6 7 8 9 10

It is now over 60 days since the last post. This thread is closed. Refresh page

Go to topic: Search the forum

top

Quick links: MUSHclient. MUSHclient help. Forum shortcuts. Posting templates. Lua modules. Lua documentation.

Information and images on this site are licensed under the Creative Commons Attribution 3.0 Australia License unless stated otherwise.