PPI - A plugin communications script

Posted by Twisol on Sun 03 Jan 2010 06:54 AM — 147 posts, 461,475 views.

USA #0
With the addition of better plugin dependency support, I decided to write a new version of LoadPPI from scratch. This version no longer has the load-on-demand functionality (because it's no longer needed and was always rather hacky), and uses a combination of CallPlugin() and MUSHclient variables to execute a method in another plugin. It also makes it so that the public interface of a plugin - the methods that the plugin wants to expose through the PPI library - doesn't have to be duplicated for every supported language, because this version doesn't load the code and execute it directly. (The PPI library itself would need to be ported for each language, but that's all.)

Hopefully some of you find this script useful. Here's an example, and at the end is the source for ppi.lua itself.

Plugin 1 (client)
local PPI = require("ppi")
OnPluginListChanged = function()
  local ppi = PPI.Load("7c08e2961c5e20e5bdbf7fc5")
  if not ppi then
    error("Dependency plugin not installed!")
  end
  
  ppi.SomeMethodHere(parameters, go, here)
end


Plugin 2 (service)
local PPI = require("ppi")

local function SomeMethodHere(parameters, go, here)
  print(parameters, go, here)
end

PPI.Expose("SomeMethodHere", SomeMethodHere)


The PPI library itself manages the intermediate steps, though at the moment it only supports strings parameters directly. It would be nice to modify it to support numbers, booleans, and tables in some manner, but it's just strings for the moment.

As promised, here is the source for ppi.lua. I wrote it in a couple hours, so it's probably not as well written as I'd like, but there it is.
local PPI_list = setmetatable({}, {__mode = "k"})

-- Takes a table and returns all values at numeric indices.
-- Example:
--  Input: {1, 2, 3, "5", "7", "10"}
--  Returns: 1, 2, 3, "5", "7", "10"
local function ExplodeParams(tbl)
  local param = table.remove(tbl, 1)
  if #tbl > 0 then
    return param, ExplodeParams(tbl)
  else
    return param
  end
end

-- A 'thunk' is a delayed resolver function.
local function new_thunk(id, func_name)
  return function(...)
    local params = {...}
    
    -- Prepare the arguments
    for i=1,#params do
      SetVariable("param" .. i .. "_" .. id, tostring(params[i]))
    end
    
    -- Call the method
    SetVariable("method_" .. id, func_name)
    CallPlugin(id, "PPI_" .. id .. "_PPI", GetPluginID())
    
    -- Clean up the arguments
    for i=1,#params do
      DeleteVariable("param" .. i .. "_" .. id)
    end
    
    -- Gather the return values
    local returns = {}
    local i = 1
    while GetPluginVariable(id, "return" .. i .. "_" .. GetPluginID()) ~= nil do
      table.insert(returns, GetPluginVariable(id, "return" .. i .. "_" .. GetPluginID()))
      i = i + 1
    end
    
    -- Have the other plugin clean up the return values
    CallPlugin(id, "PPI_" .. id .. "_PPI_CLEAN", GetPluginID())
    
    return ExplodeParams(returns)
  end
end

-- If the requested function hasn't yet had a thunk created,
-- create a new thunk and return it.
local PPI_meta = {
  __index = function(tbl, idx)
    local thunk = new_thunk(PPI_list[tbl].id, idx)
    tbl[idx] = thunk
    return thunk
  end,
}

local PPI = {
  -- Used to retreive a PPI for a specified plugin.
  Load = function(plugin_id)
    if not IsPluginInstalled(plugin_id) then
      return false
    end
    
    local tbl = PPI_list[plugin_id]
    if not tbl then
      tbl = setmetatable({}, PPI_meta)
      PPI_list[tbl] = {id = plugin_id}
      PPI_list[plugin_id] = tbl
    end
    return tbl
  end,
  
  -- Used by a plugin to expose methods to other plugins
  -- through its own PPI.
  Expose = function(name, func)
    local myPPI = PPI_list[GetPluginID()]
    myPPI[name] = func
  end,
}

-- create a PPI for this plugin
myPPI = {}
PPI_list[myPPI] = {id = GetPluginID()}
PPI_list[GetPluginID()] = myPPI

-- PPI request resolver
_G["PPI_" .. GetPluginID() .. "_PPI"] = function(id)
  local myPPI = PPI_list[GetPluginID()]
  local myID = PPI_list[myPPI].id
  if not myPPI then
    return
  end
  
  local params = {}
  local i = 1
  while GetPluginVariable(id, "param" .. i .. "_" .. myID) ~= nil do
    table.insert(params, GetPluginVariable(id, "param" .. i .. "_" .. myID))
    i = i + 1
  end
  
  local func_name = GetPluginVariable(id, "method_" .. myID)
  local func = myPPI[func_name]
  if not func then
    return
  end
  
  local returns = {func(ExplodeParams(params))}
  for i=1, #returns do
    SetVariable("return" .. i .. "_" .. id, tostring(returns[i]))
  end
end

-- Return value cleaner
_G["PPI_" .. GetPluginID() .. "_PPI_CLEAN"] = function(id)
  local i = 1
  while GetVariable("return" .. i .. "_" .. id) ~= nil do
    DeleteVariable("return" .. i .. "_" .. id)
    i = i + 1
  end
end

return PPI
Amended on Sun 03 Jan 2010 07:01 AM by Twisol
Australia Forum Administrator #1
I'm starting to play with this. I think you can delete these lines:


-- Takes a table and returns all values at numeric indices.
-- Example:
--  Input: {1, 2, 3, "5", "7", "10"}
--  Returns: 1, 2, 3, "5", "7", "10"
local function ExplodeParams(tbl)
  local param = table.remove(tbl, 1)
  if #tbl > 0 then
    return param, ExplodeParams(tbl)
  else
    return param
  end
end


Replace calls to ExplodeParams by a call to unpack. That seems to work.

http://www.gammon.com.au/scripts/doc.php?lua=unpack

You can also simplify the Expose function by looking up the function yourself, like this:


 -- Used by a plugin to expose methods to other plugins
  -- through its own PPI.
  Expose = function (name)
    assert (type (_G [name]) == "function", "Function " .. name .. " does not exist.")
    PPI_list[GetPluginID()] [name] = _G [name]
  end,


That way instead of:


PPI.Expose("SomeMethodHere", SomeMethodHere)


you can use:


PPI.Expose "SomeMethodHere"


However "SomeMethodHere" now has to be a global function and not a local one. However that saves duplication of typing, and possibly mistyping the function name.
Australia Forum Administrator #2
I offer this up as an enhanced version of your PPI system. Basically it improves upon it by keeping the types of the supplied parameters, and the returned values.


module (..., package.seeall)

require "serialize"

local PPI_list = {}

-- A 'thunk' is a delayed resolver function.
local function new_thunk(id, func_name)
  return function(...)
    
    -- Prepare the arguments
    SetVariable("params_" .. id, serialize.save_simple {...})
    
    -- Call the method
    SetVariable("method_" .. id, func_name)
    CallPlugin(id, "PPI_" .. id .. "_PPI", GetPluginID())
    
    -- Clean up the arguments
    DeleteVariable("params_" .. id)
    
    local returns = {}  -- table of returned values
    local r = GetPluginVariable(id, "returns_" .. GetPluginID())
    if r then 
      local f = assert (loadstring ("r = " .. r))  -- convert serialized data back
      setfenv (f, returns)  -- make sure we use the returns table as our destination
      f ()   -- put data back in table "r"
    end -- if
    
    -- Have the other plugin clean up the return values
    CallPlugin(id, "PPI_" .. id .. "_PPI_CLEAN", GetPluginID())
    
    return unpack (returns.r)
  end
end

-- If the requested function hasn't yet had a thunk created,
-- create a new thunk and return it.
local PPI_meta = {
  __index = function(tbl, idx)
    local thunk = new_thunk(PPI_list[tbl].id, idx)
    tbl[idx] = thunk
    return thunk
  end,
}

local PPI = {
  -- Used to retreive a PPI for a specified plugin.
  Load = function(plugin_id)
    if not IsPluginInstalled(plugin_id) then
      return false
    end
    
    local tbl = PPI_list[plugin_id]
    if not tbl then
      tbl = setmetatable({}, PPI_meta)
      PPI_list[tbl] = {id = plugin_id}
      PPI_list[plugin_id] = tbl
    end
    return tbl
  end,
  
  -- Used by a plugin to expose methods to other plugins
  -- through its own PPI.
  Expose = function (name)
    assert (type (_G [name]) == "function", "Function " .. name .. " does not exist.")
    PPI_list[GetPluginID()] [name] = _G [name]
  end,
}

-- create a PPI for this plugin
myPPI = {}
PPI_list[myPPI] = {id = GetPluginID()}
PPI_list[GetPluginID()] = myPPI

-- PPI request resolver
_G["PPI_" .. GetPluginID() .. "_PPI"] = function(id)
  local myPPI = PPI_list[GetPluginID()]
  local myID = PPI_list[myPPI].id
  if not myPPI then
    return
  end
  
  local params = {}  -- table of parameters
  local p = GetPluginVariable(id, "params_" .. myID)
  if p then 
    local f = assert (loadstring ("t = " .. p))  -- convert serialized data back
    setfenv (f, params)  -- make sure we use the params table as our destination
    f ()   -- put data back in table "t"
  end -- if
    
  local func_name = GetPluginVariable(id, "method_" .. myID)
  local func = myPPI[func_name]
  if not func then
    return
  end
  
  -- call target function, get return values, serialize back into variable
  SetVariable("returns_" .. id, serialize.save_simple {func(unpack (params.t))})

end -- PPI request resolver

-- Return value cleaner
_G["PPI_" .. GetPluginID() .. "_PPI_CLEAN"] = function(id)
  DeleteVariable("returns_" .. id)
end

return PPI


For example, my service is:


local PPI = require("ppi")

require "tprint"

function SomeMethodHere (...)
  tprint {...}  
  
  return 1, "x", false, { y = 88 }
end

PPI.Expose "SomeMethodHere"


And the client is:


local PPI = require("ppi")
OnPluginListChanged = function()
  ppi = PPI.Load("15783160bde378741f9652d1")
  if not ppi then
    error("Dependency plugin not installed!")
  end
  tprint {ppi.SomeMethodHere (42, "Nick", true, {a = "56"} )}
end


Notice that the client sends various types to the service (number, string, boolean, table) and the service replies with number, string, boolean, table.



Client received:

1=42
2="Nick"
3=true
4:
  "a"="56"

Service replied:

1=1
2="x"
3=false
4:
  "y"=88


Note how the service received the correct types, and the client received back the correct types.

This version is slightly simpler, if anything, because of instead of lots of MUSHclient variables being set up, a single variable is created with serialize.save_simple.
Australia Forum Administrator #3
Since you can send tables, you could call a function in another plugin and send stuff like style runs, for example, or an inventory list.
Australia Forum Administrator #4
I don't know about using the weak keys, either. You may find that your table gets garbage-collected when you least expect it.
USA #5
Briefly, because I'm still reading through it, I want to mention that one of the key reasons I moved to CallPlugin() was because then only the PPI library would need to be ported between languages. Using the 'serialize' library would bind it to Lua, or require compatible serialization libraries in other languages.

I was planning on writing a PPI-specific notation that would be attached to each parameter, something like "n|100", "s|a string", "b|0" (boolean), and for tables I wasn't quite sure, but I was probably going to use the Array* methods and convert the table into a string, like "t|the array stuff here".


About the weak keys, you're probably right.


Also, I really appreciate that you took the time to play around with it. :)
Amended on Thu 07 Jan 2010 11:45 PM by Twisol
USA #6
As a side note, I somewhat preferred the two-argument version of Expose, as you could map a function to a string with a different name, or inline the function right there:

PPI.Expose("MyMethod", function()
    print("Hallos.")
  end
)


Basically it's just extra flexibility. It probably shouldn't be too hard to write a version that accepts either one or two arguments, though.
Australia Forum Administrator #7
Twisol said:

Briefly, because I'm still reading through it, I want to mention that one of the key reasons I moved to CallPlugin() was because then only the PPI library would need to be ported between languages. Using the 'serialize' library would bind it to Lua, or require compatible serialization libraries in other languages.


OK, well as it stands, the module requires to be used from Lua, so using the Lua serialization makes it more powerful for that purpose.

If someone did (say) a VBscript version then they could handle the different data types differently, but for a pure Lua application, I think it is easier and more natural than having to pass down data types in the way you describe.

I would suggest that modern developers are probably going to use Lua, and if so they might have a helper plugin (also in Lua) and communicate with it using your module. However if they insist on using something else, then they can take your broad ideas and make something similar for their own language.
USA #8
It's not really so difficult. I'm working on an improved version currently (I did incorporate many/all of your suggestions), and it's coming along fairly well. At this stage, it's using the Array*() functions to load all of the parameters into, then exporting it into a single MUSH variable. I have a promising plan to extend this to table parameters as well (including nested tables), by giving each table its own MUSH variable, and using "t|2" for the table-type in the array, where 2 is (part of) the ID of the MUSH variable containing that table's definition.

It's not much trouble, and I enjoy this kind of critical thinking anyways. I hope to post a mid-way version soon.
Amended on Fri 08 Jan 2010 01:21 AM by Twisol
USA #9
Alright, here's a midway version, switched up to use arrays for serialization. It still restricts parameters to strings, but that should change soon.

It also supports both one-arg and two-args versions of PPI.Expose. If the second argument is missing, it simply uses _G[first argument].

local PPI_list = {}

local myID = GetPluginID()
local myPPI = {}

local params_id  = function(id) return "PPIparams_" .. id  end
local method_id  = function(id) return "PPImethod_" .. id  end
local returns_id = function(id) return "PPIreturns_" .. id end

local request_id = function(id) return "PPI_" .. id .. "_REQUEST" end
local cleanup_id = function(id) return "PPI_" .. id .. "_CLEANUP" end


local function request(id, func_name, ...)
  -- Prepare the arguments
  local params = {...}
  
  ArrayCreate(params_id(id))
  
  for i=1,#params do
    ArraySet(params_id(id), tostring(i), tostring(params[i]))
  end

  SetVariable(params_id(id), ArrayExport(params_id(id), "|"))
  ArrayDelete(params_id(id))
  
  -- Call the method
  SetVariable(method_id(id), func_name)
  CallPlugin(id, request_id(id), myID)
  
  -- Clean up the arguments
  DeleteVariable(params_id(id))
  DeleteVariable(method_id(id))
  
  -- Gather the return values
  local returns = {}
  
  ArrayCreate(returns_id(id))
  ArrayImport(returns_id(id), GetPluginVariable(id, returns_id(myID)), "|")
  
  for k,v in pairs(ArrayList(returns_id(id))) do
    returns[tonumber(k)] = v
  end
  ArrayDelete(returns_id(id))
  
  -- Have the other plugin clean up its return values
  CallPlugin(id, cleanup_id(id), myID)
  
  return unpack(returns)
end

-- A 'thunk' is a delayed resolver function.
local function new_thunk(id, func_name)
  return function(...)
    return request(id, func_name, ...)
  end
end

-- If the requested function hasn't yet had a thunk created,
-- create a new thunk and return it.
local PPI_meta = {
  __index = function(tbl, idx)
    local thunk = new_thunk(PPI_list[tbl].id, idx)
    tbl[idx] = thunk
    return thunk
  end,
}

local PPI = {
  -- Used to retreive a PPI for a specified plugin.
  Load = function(plugin_id)
    if not IsPluginInstalled(plugin_id) then
      return false
    end
    
    local tbl = PPI_list[plugin_id]
    if not tbl then
      tbl = setmetatable({}, PPI_meta)
      PPI_list[tbl] = {id = plugin_id}
      PPI_list[plugin_id] = tbl
    end
    return tbl
  end,
  
  -- Used by a plugin to expose methods to other plugins
  -- through its own PPI.
  Expose = function(name, func)
    myPPI[name] = (func and func or _G[name])
  end,
}

-- PPI request resolver
_G[request_id(myID)] = function(id)
  ArrayCreate(params_id(id))
  ArrayImport(params_id(id), GetPluginVariable(id, params_id(myID)), "|")
  
  local params = {}
  for k,v in pairs(ArrayList(params_id(id))) do
    params[tonumber(k)] = v
  end
  ArrayDelete(params_id(id))
  
  local func_name = GetPluginVariable(id, method_id(myID))
  local func = myPPI[func_name]
  if not func then
    return
  end
  
  local returns = {func(unpack(params))}
  ArrayCreate(returns_id(id))
  
  for i=1,#returns do
    ArraySet(returns_id(id), tostring(i), tostring(returns[i]))
  end

  SetVariable(returns_id(id), ArrayExport(returns_id(id), "|"))
  ArrayDelete(returns_id(id))
end

-- Return value cleaner
_G[cleanup_id(myID)] = function(id)
  DeleteVariable(returns_id(id))
end

return PPI
USA #10
Once 4.46 is released, I will probably build support in for checking a plugin's nonce value [1], so that the whole process is fairly automatic.

[1] http://www.gammon.com.au/forum/?id=9969&page=999
Australia Forum Administrator #11
While you were doing that I did another version myself. ;)

This one removes your PPI_list table, which I couldn't really see the purpose of. It is somewhat shorter as it effectively just uses an upvalue to store the only thing you really need to remember, which is the function associated with the name.


--[[

Author:  	Twisol 
Date:  3rd January 2010

Amendments: Nick Gammon
Date: 8th January 2010

Example of use:

SERVICE

-- require PPI module
require "ppi"

-- exposed function
function SomeMethodHere (a, b, c, d)
  -- do something with a, b, c, d
  return 1, 2, 3, 4
end

-- notify PPI of this function
ppi.Expose "SomeMethodHere"


CLIENT

-- require PPI module
require "ppi"

-- resolve dependencies
function OnPluginListChanged ()
  
  -- get PPI entries for all exposed function in this plugin
  my_service = ppi.Load ("15783160bde378741f9652d1")  -- plugin ID of service plugin

  if not my_service then
    Note ("Dependency plugin not installed!")
  end
  
end -- OnPluginListChanged

-- later on in plugin ...

-- call SomeMethodHere in other plugin, passing various data types, getting results

if my_service then
  w, x, y, z = my_service.SomeMethodHere (42, "Nick", true, { a = 63, b = 22 } )
end -- if service installed

--]]

module (..., package.seeall)

require "serialize"

-- For any function in our PPI table, try to call that in the target plugin
local PPI_meta = {
  __index = function(tbl, idx)
    return function(...)
       
        -- Call the method in the target plugin
        local status = CallPlugin (tbl._id, "PPI_" .. idx .. "_PPI_", serialize.save_simple {...})
        
        -- explain a bit if we failed
        if status ~= error_code.eOK then
          ColourNote ("white", "red", "Error calling " .. idx .. " using PPI from " .. GetPluginName ())
          check (status)
        end -- if
        
        local returns = {}  -- table of returned values
        local r = GetPluginVariable(tbl._id, "PPI_returns_PPI_") or "{}"
        local f = assert (loadstring ("r = " .. r))  -- convert serialized data back
        setfenv (f, returns) () -- load the returned values into 'returns'
        
        return unpack (returns.r)
      end  -- generated function
    end,
}  -- end PPI_meta table

-- PPI request resolver
local function PPI_resolver (func) 
  return function (p)
    local params = {}  -- table of parameters
    local f = assert (loadstring ("t = " .. p))  -- convert serialized data back
    setfenv (f, params) ()  -- load the parameters into 'params'
    
    -- call target function, get return values, serialize back into variable
    SetVariable("PPI_returns_PPI_", serialize.save_simple {func(unpack (params.t))})
  end -- generated function
  
end -- PPI_resolver

-- EXPOSED FUNCTIONS

-- We "load" a plugin by checking it exists, and creating a table saving the
-- target plugin ID, and have a metatable which will handle function calls
function Load (plugin_id)
  if IsPluginInstalled (plugin_id) then
    return setmetatable ({ _id = plugin_id }, PPI_meta)
  end -- if plugin exists
end -- function Load 

-- Used by a plugin to expose methods to other plugins
-- Each exposed function will be added to global namespace as PPI_<name>_PPI_
function Expose (name, func)
  assert (type (func or _G [name]) == "function", "Function " .. name .. " does not exist.")
  _G ["PPI_" .. name .. "_PPI_"] = PPI_resolver (func or _G [name])
end -- function Expose

Australia Forum Administrator #12
I should point out that testing shows that this error becomes unrecoverable, without reinstalling *this* plugin:


OnPluginListChanged = function()
  local ppi = PPI.Load("7c08e2961c5e20e5bdbf7fc5")
  if not ppi then
    error("Dependency plugin not installed!")
  end
  
  ppi.SomeMethodHere(parameters, go, here)
end


The error raised in OnPluginListChanged makes MUSHclient flag OnPluginListChanged as "a module in error" and no longer calls it, even if the plugin list changes again.

Thus, reinstalling (or installing for the first time) the missing plugin, is not sufficient to recover.

You may want to downgrade that to a warning (eg. Note instead of error), so that if the other plugin is subsequently installed, OnPluginListChanged will be called again to do the PPI.Load.
USA #13
(Currently working on table serialization; the other three datatypes work beautifully)

On ppi_list: Well, it does provide a good place to store "private" values like the plugin's ID, as well as the current 'nonce' for the PPI in order to implement the nonce-checking I mentioned before.

On Note() versus error(): It depends on the plugin, really. If it will work acceptably even if the dependency isn't there, sure, a Note() is probably better. But if you absolutely don't want the plugin doing anything at all if the dependency isn't loaded, if things such as collecting information or other stuff are pointless and wasteful if the dependency isn't there, then I would say an error() is the way to go. You can always have the error say to reinstall the plugin, after all.
Amended on Fri 08 Jan 2010 03:48 AM by Twisol
USA #14
Alrighty, here's the more-or-less final version of PPI. It now supports string, number, boolean, and table values, and string and number keys. Some scripting languages can only support one or the other as keys; in these cases, ports of PPI may default to strings, or come up with a language-specific mechanism. So long as the shared interface between PPI libraries is unchanged, the implementation details aren't terribly important.

Nick's previous service/client examples should also be valid under this version of PPI. I've added version numbers to the PPI table to be safe; this is v1.0.0.

(Due to post size limits, I had to offload the library to a pastebin: [1])

[1] http://mushclient.pastebin.com/f7b5b37a9
Amended on Fri 08 Jan 2010 10:07 AM by Twisol
USA #15
I acceidentally left off the code for cleaning up after the request, heh. Re-pasted and post edited, if by some miracle someone downloaded it before I fixed it, please download it again.
USA #16
New version v1.0.1, fixing the clean-up of variables I somehow missed, as well as fixing the array ID deserialize() would use so it would be unique within a given deserialization.

local __V_MAJOR, __V_MINOR, __V_PATCH = 1, 0, 1
local __VERSION = string.format("%d.%d.%d", __V_MAJOR, __V_MINOR, __V_PATCH)
 
local PPI_list = {}
 
local myID = GetPluginID()
local myPPI = {}
 
local array_id   = function(id)      return "PPIarray_" .. id   end
local params_id  = function(id)      return "PPIparams_" .. id  end
local method_id  = function(id)      return "PPImethod_" .. id  end
 
local request_id = function(id) return "PPI_" .. id .. "_REQUEST" end
local cleanup_id = function(id) return "PPI_" .. id .. "_CLEANUP" end
 
 
function serialize(params, params_list)
  if not params_list then
    local params_list = {}
    serialize(params, params_list)
   
    local list = {}
    for k,v in ipairs(params_list) do
      list[k] = v
    end
    return list
  end
 
  if params_list[params] then
    return params_list[params]
  end
 
  local index = #params_list + 1
  params_list[params] = index
  params_list[index] = true
 
  local id = array_id(index)
  ArrayCreate(id)
 
  for k,v in pairs(params) do
    local key = nil
    if type(k) == "string" then
      key = "s:" .. k
    elseif type(k) == "number" then
      key = "n:" .. k
    end
   
    if key then
      local value = "z:~"
     
      if type(v) == "string" then
        value = "s:" .. v
      elseif type(v) == "number" then
        value = "n:" .. v
      elseif type(v) == "boolean" then
        value = "b:" .. (v and "1" or "0")
      elseif type(v) == "table" then
        value = "t:" .. serialize(v, params_list)
      end
     
      ArraySet(id, key, value)
    end
  end
 
  params_list[index] = ArrayExport(id, "|")
  ArrayDelete(id)
 
  return index
end
 
function deserialize(data_list, index, state)
  if not index or not state then
    return deserialize(data_list, 1, {})
  end
 
  if state[index] then
    return state[index]
  end
 
  local tbl = {}
  state[index] = tbl
 
  local id = array_id(index)
  ArrayCreate(id)
  ArrayImport(id, data_list[index], "|")
 
  for k,v in pairs(ArrayList(id)) do
    local key_type = k:sub(1,1)
    local key = nil
   
    if key_type == "s" then
      key = k:sub(3)
    elseif key_type == "n" then
      key = tonumber(k:sub(3))
    end
   
    if key then
      local item_type = v:sub(1,1)
      local item = v:sub(3)
     
      if item_type == "s" then
        tbl[key] = item
      elseif item_type == "n" then
        tbl[key] = tonumber(item)
      elseif item_type == "b" then
        tbl[key] = ((item == "1") and true or false)
      elseif item_type == "t" then
        tbl[key] = deserialize(data_list, tonumber(item), state)
      else
        tbl[key] = nil
      end
    end
  end
 
  ArrayDelete(id)
 
  return tbl
end
 
local function request(id, func_name, ...)
  -- Prepare the arguments
  local params = {...}
  for k,v in ipairs(serialize(params)) do
    SetVariable(params_id(id) .. "_" .. k, v)
  end
 
  -- Call the method
  SetVariable(method_id(id), func_name)
  CallPlugin(id, request_id(id), myID)
 
  -- Clean up the arguments
  for i=1,#params do
    DeleteVariable(params_id(id) .. "_" .. i)
  end
  DeleteVariable(method_id(id))
 
  -- Gather the return values
  local returns = {}
  local i = 1
  while GetPluginVariable(id, params_id(myID) .. "_" .. i) do
    returns[i] = GetPluginVariable(id, params_id(myID) .. "_" .. i)
    i = i + 1
  end
  returns = deserialize(returns)
 
  -- Have the other plugin clean up its return values
  CallPlugin(id, cleanup_id(id), myID)
 
  return unpack(returns)
end
 
-- A 'thunk' is a delayed resolver function.
local function new_thunk(id, func_name)
  return function(...)
    return request(id, func_name, ...)
  end
end
 
-- If the requested function hasn't yet had a thunk created,
-- create a new thunk and return it.
local PPI_meta = {
  __index = function(tbl, idx)
    local thunk = new_thunk(PPI_list[tbl].id, idx)
    tbl[idx] = thunk
    return thunk
  end,
}
 
local PPI = {
  __V = __VERSION,
  __V_MAJOR = __V_MAJOR,
  __V_MINOR = __V_MINOR,
  __V_PATCH = __V_PATCH,
 
  -- Used to retreive a PPI for a specified plugin.
  Load = function(plugin_id)
    if not IsPluginInstalled(plugin_id) then
      return false
    end
   
    local tbl = PPI_list[plugin_id]
    if not tbl then
      tbl = setmetatable({}, PPI_meta)
      PPI_list[tbl] = {id = plugin_id}
      PPI_list[plugin_id] = tbl
    end
    return tbl
  end,
 
  -- Used by a plugin to expose methods to other plugins
  -- through its own PPI.
  Expose = function(name, func)
    myPPI[name] = (func and func or _G[name])
  end,
}
 
-- PPI request resolver
_G[request_id(myID)] = function(id)
  -- Get requested method
  local func = myPPI[GetPluginVariable(id, method_id(myID))]
  if not func then
    return
  end
 
  -- Deserialize parameters
  local params = {}
  local i = 1
  while GetPluginVariable(id, params_id(myID) .. "_" .. i) do
    params[i] = GetPluginVariable(id, params_id(myID) .. "_" .. i)
    i = i + 1
  end
  params = deserialize(params)
 
  -- Call method, return values
  local returns = {func(unpack(params))}
  for k,v in ipairs(serialize(returns)) do
    SetVariable(params_id(id) .. "_" .. k, v)
  end
end
 
-- Return value cleaner
_G[cleanup_id(myID)] = function(id)
  local i = 1
  while GetVariable(params_id(id) .. "_" .. i) do
    DeleteVariable(params_id(id) .. "_" .. i)
    i = i + 1
  end
end
 
return PPI
Australia Forum Administrator #17
Twisol said:

On Note() versus error(): It depends on the plugin, really. If it will work acceptably even if the dependency isn't there, sure, a Note() is probably better. But if you absolutely don't want the plugin doing anything at all if the dependency isn't loaded, if things such as collecting information or other stuff are pointless and wasteful if the dependency isn't there, then I would say an error() is the way to go. You can always have the error say to reinstall the plugin, after all.


True, it depends on the situation. However I think showing something like this to the end-user is just confusing for them:


Run-time error
Plugin: ppi_test_client (called from world: SmaugFUSS)
Function/Sub: OnPluginListChanged called by Plugin ppi_test_client
Reason: Executing plugin ppi_test_client sub OnPluginListChanged
[string "Plugin"]:9: Dependency plugin not installed!
stack traceback:
        [C]: in function 'error'
        [string "Plugin"]:9: in function <[string "Plugin"]:3>
Error context in script:
   5 :   my_service = ppi.Load("15783160bde378741f9652d1")
   6 :   if my_service then
   7 :     tprint (my_service)
   8 :   else
   9*:     error("Dependency plugin not installed!")
  10 :   end
  11 :   
  12 : end
  13 : 


They will think they did something wrong, or the plugin is faulty, and you will get a forum query.

Better to display something like:


This plugin requires you to install: Twisols_Stats_Gather plugin.

This is available from: www.somewebsite.com

Please download and install that, and then try again.


As for the plugin not really working if the dependency is not available, one option is to simple disable itself, eg.


EnablePlugin(GetPluginID (), false)


Then callbacks won't be called, and triggers and aliases are disabled.

A problem with that still is that the disabled plugin won't process OnPluginListChanged still (when the dependency becomes available).

Another option is to put all triggers etc. into a group, and just disable the group.

Another option is to just check at applicable places if the dependency plugin is there, eg.


if my_service then
  my_service.DoSomethingHere (a, b, c)
end -- if plugin available


The last case could be useful if the plugin is not essential, for example, it plays sounds.
Amended on Fri 08 Jan 2010 08:45 PM by Nick Gammon
USA #18
True, I like the idea of a Note() then disabling itself. I don't see the requirement of reinstalling the plugin afterward to be a huge deal, personally.

Putting all triggers, aliases, etc. into one group means that they can't be part of any other group. That's not really a good thing when you have sets triggers that depend on other triggers.

The "check if dependency, if so, do something" approach is probably best for soft dependencies, where the plugin can keep working if its dependency isn't loaded. No complaints here.
USA #19
I'm proud to release PPI v1.1.0, with one minor fix and two additions to make it easier and more natural to use. [1]

One change is the addition of a SUPPORTS internal message, which queries the service about the existence of a particular message. This is utilized in __index, so when you ask for a nonexistant function, it returns nil, just like it would natively.

Second is the addition of (nil, "error string") return values to PPI.Load(). It first checks to see if the plugin is installed (if not, it returns nil, "not_installed"), then it checks if the plugin supports PPI (if not, it returns nil, "no_ppi").


Also, Nick, I noticed you amended the 4.46 release notes to show that you added your derived version of PPI to the standard distribution! I'm honored that my work was found to be so useful. I do feel like pointing out, though, that this latest version (and indeed the one before, IIRC) supports deeper serialization, where a table can refer to other tables or even itself. I also think that this version is perhaps a bit less limiting, as yours (I assume) seems to reserve keys beginning with _, like _id and _version.

Incidentally, I'm considering adding new functionality that would support directly accessing (but not modifying) exposed values like strings, numbers, booleans, and tables, as well as what we already have, methods. I may also be able to add "serialization" functionality for passed/returned methods, by storing them in another section of the private data of the PPI, passing an internal name/value referring to it, and on the other side creating an internal thunk that is passed back to the user.

[1] http://mushclient.pastebin.com/f75922737
Australia Forum Administrator #20
I think my version has the benefit of simplicity, whereas yours has the benefit of greater power. Perhaps both can be part of the next release, although we need a new name for one of them.
Amended on Sat 09 Jan 2010 05:01 AM by Nick Gammon
USA #21
I don't know... they're both very simple on the user side of things. They have the exact same user interface, at any rate. It's just how things are done internally that's different, and lends it greater power. And it's how things appear to the user that's important, right? Mine also does things in a certain way to make it feel as native as possible, like checking if a method is supported and returning nil if not, which I don't think yours does.

It seems kind of strange to provide two libraries that do the same thing, only one is named differently and one does things in a better and more flexible (IMHO) way.

Of course, if you have any suggestions for simplifying my library's design, I'm all for hearing them! I always try to make my work as clean and complete as possible.
Australia Forum Administrator #22
OK, well for the record, here is my proposed release version:


--[[

PLUGIN-to-PLUGIN-INTERFACE (PPI)

Author:  	Twisol 
Date:  3rd January 2010

Amendments: Nick Gammon
Date: 8th January 2010

Example of use:

SERVICE

-- require PPI module
require "ppi"

-- exposed function
function SomeMethodHere (a, b, c, d)
  -- do something with a, b, c, d
  return 1, 2, 3, 4
end

-- notify PPI of this function
ppi.Expose "SomeMethodHere"

-- Or, for anonymous functions:
ppi.Expose ("DoSomethingElse", function () print "hello" end)

CLIENT

-- require PPI module
require "ppi"

-- resolve dependencies
function OnPluginListChanged ()
  
  -- get PPI entries for all exposed function in this plugin
  my_service = ppi.Load ("15783160bde378741f9652d1")  -- plugin ID of service plugin

  if not my_service then
    Note ("Dependency plugin not installed!")
  end
  
end -- OnPluginListChanged

-- later on in plugin ...

-- call SomeMethodHere in other plugin, passing various data types, getting results

if my_service then
  w, x, y, z = my_service.SomeMethodHere (42, "Nick", true, { a = 63, b = 22 } )
end -- if service installed

NOTES
-----

ppi.Load returns a table with various values in it about the target plugin (see below
for what they are). For example, _name is the plugin name of the target plugin, and
_version is the version number of that plugin.

If ppi.Load returns no value (effectively, nil) then the target plugin was not installed.

Provided a non-nil result was returned, you can then call any exposed function in the 
target plugin. There is currently no mechanism for finding what functions are exposed, for
simplicity's sake. However it would be possible to make a service function that returned all
exposed functions. If service plugins evolve in functionality, checking the target plugin's
version (the _version variable) should suffice for making sure plugins are synchronized.

To avoid clashes in variable names, you cannot expose a function starting with an underscore.

Communication with the target plugin is by global variables set up by the Expose function, along
the lines of:

PPI_function_name_PPI_  (one for each exposed function)

Also:

PPI__returns__PPI_ is used for storing the returned values.

--]]

-- hide all except non-local variables
module (..., package.seeall)

-- for transferring variables
require "serialize"

-- PPI version
local V_MAJOR, V_MINOR, V_PATCH = 1, 1, 0
local VERSION = string.format ("%d.%d.%d", V_MAJOR, V_MINOR, V_PATCH)

-- called plugin uses this variable to store returned values
local RETURNED_VALUE_VARIABLE = "PPI__returns__PPI_"

-- For any function in our PPI table, try to call that in the target plugin
local PPI_meta = {
  __index = function (tbl, idx)
    if (idx:sub (1, 1) ~= "_") then
      return function(...)
          -- Call the method in the target plugin
          local status = CallPlugin (tbl._id, "PPI_" .. idx .. "_PPI_", serialize.save_simple {...})
          
          -- explain a bit if we failed
          if status ~= error_code.eOK then
            ColourNote ("white", "red", "Error calling " .. idx .. 
                        " in plugin " .. tbl._name .. 
                        " using PPI from " .. GetPluginName () ..
                        " (" .. error_desc [status] .. ")")
            check (status)
          end -- if
          
          -- call succeeded, get any returned values
          local returns = {}  -- table of returned values
          local s = GetPluginVariable(tbl._id, RETURNED_VALUE_VARIABLE) or "{}"
          local f = assert (loadstring ("t = " .. s))  -- convert serialized data back
          setfenv (f, returns) () -- load the returned values into 'returns'
          
          -- unpack returned values to caller
          return unpack (returns.t)
        end  -- generated function
      end -- not starting with underscore
    end  -- __index function
}  -- end PPI_meta table

-- PPI request resolver
local function PPI_resolver (func) 
  return function (s)  -- calling plugin serialized parameters into a single string argument
    local params = {}  -- table of parameters
    local f = assert (loadstring ("t = " .. s))  -- convert serialized data back
    setfenv (f, params) ()  -- load the parameters into 'params'
    
    -- call target function, get return values, serialize back into variable
    SetVariable(RETURNED_VALUE_VARIABLE, serialize.save_simple {func(unpack (params.t))})
  end -- generated function
  
end -- PPI_resolver

-- EXPOSED FUNCTIONS

-- We "load" a plugin by checking it exists, and creating a table saving the
-- target plugin ID etc., and have a metatable which will handle function calls
function Load (plugin_id)
  if IsPluginInstalled (plugin_id) then
    return setmetatable (
      { _id = plugin_id,   -- so we know which plugin to call
        _name = GetPluginInfo (plugin_id, 1),
        _author = GetPluginInfo (plugin_id, 2),
        _filename = GetPluginInfo (plugin_id, 6),
        _enabled = GetPluginInfo (plugin_id, 17),
        _version = GetPluginInfo (plugin_id, 18),
        _required_version = GetPluginInfo (plugin_id, 19),
        _directory = GetPluginInfo (plugin_id, 20),
        _PPI_V_MAJOR = V_MAJOR,  -- version info
        _PPI_V_MINOR = V_MINOR,
        _PPI_V_PATCH = V_PATCH,
        _PPI_VERSION = VERSION,
       }, 
       PPI_meta)  -- everything except the above will generate functions
  else
    return nil, "Plugin ID " .. plugin_id .. " not installed"  -- in case you assert
  end -- if 
end -- function Load 

-- Used by a plugin to expose methods to other plugins
-- Each exposed function will be added to global namespace as PPI_<name>_PPI_
function Expose (name, func)
  assert (type (func or _G [name]) == "function", "Function " .. name .. " does not exist.")
  _G ["PPI_" .. name .. "_PPI_"] = PPI_resolver (func or _G [name])
end -- function Expose


I liked your idea of incorporating the PPI version, so I added that. Changes you might consider are:

  • The "module" line hides all local variables from the rest of the script (the calling script) so there aren't any possible clashes between module variables and script variables.
  • In the Load function, if the plugin does not exist, I return nil followed by an error message. This makes Load suitable to assert, eg.

    
    my_service = assert (ppi.Load "<something>")
    


    Of course, this goes against what I was saying about a helpful error message, but if you are going to just do an error anyway, this lets you do it in a single line.
  • In the table returned by Load I use GetPluginInfo to return various useful variables, like the plugin's name and version. The version helps for version checks of the plugin, and the name helps in doing error messages.
  • I didn't bother cleaning up the returned value variables, on the grounds that this is an overhead, and you will probably be doing it thousands of times (creating the variable / deleting it) if the service is used a lot.


Between us we are hopefully moving towards something truly elegant and useful. :-)
USA #23
Actually, module() isn't needed to hide local variables. A file is loaded and then executed as though it were a function, just like loadstring("script")(). All module() does is set the environment of the currently executing frame - that would be the file - to a new table, which is also set into packages.loaded[module_name_here]. It doesn't actually do anything to local values, precisely because they are local to that frame anyways.

With Load, I also return nil followed by an error message, though my messages are shorter and more useful to the calling script rather than the user (which makes sense, because the plugin will want to give its own error message).

In regards to the PPI returned from PPI.Load, I have to admit that it makes me uncomfortable to be imposing restrictions on what names the service can expose. As you've shown, most of that information can be easily accessed through GetPluginInfo, and most of it (directory, required_version) won't be used very often at all. 'enabled' is even worse, in my opinion, as it isn't updated automatically when the plugin is disabled. The PPI version numbers, well... to be honest, I have no idea why you're including them in each individual PPI rather than the package table itself. It might make sense to store the service's PPI version, but not the client's.

On reflection, I agree with the cleanup overhead point to a certain extent. However, I ran a test loop to check how long it takes to execute 20,000 total SetVariable() and DeleteVariable() calls (that's 10,000 of each), and it took less than a second. I don't think we have to worry about it too much, and I do prefer to keep things tidy.


Writing the previously-mentioned function-passing and direct indexing of exposed values is turning out to be a little difficult, but mostly because of assumptions I had made at the beginning. It will probably take a little longer, but I'm cleaning up as I go anyways, so it will be better in the long run.

I must agree, this is a very fun project and I think it is astonishingly useful already. Could I ask if you're planning on releasing 4.46 soon? I'd like to finish these last additions beforehand if I have enough time.


EDIT: In regards to module() again, you're right in that it would hide the serialize library, since that sets its package into _G. A benefit of mine, though, is that the serialization routines are part of the PPI script already, so only exactly what I want is exposed, and no more.
Amended on Sat 09 Jan 2010 08:21 PM by Twisol
Australia Forum Administrator #24
Twisol said:

Could I ask if you're planning on releasing 4.46 soon? I'd like to finish these last additions beforehand if I have enough time.


Don't panic, I'll wait until all this is stabilized.


Twisol said:

EDIT: In regards to module() again, you're right in that it would hide the serialize library, since that sets its package into _G. A benefit of mine, though, is that the serialization routines are part of the PPI script already, so only exactly what I want is exposed, and no more.


I'm not sure if this will clash if someone requires "serialize" for serializing their normal variables. Why not make them local variables? And why not, again, use the module function? I can't see a major objection to it.
Australia Forum Administrator #25
Twisol said:

With Load, I also return nil followed by an error message, though my messages are shorter and more useful to the calling script rather than the user (which makes sense, because the plugin will want to give its own error message).


Not in the version posted earlier on this page.
USA #26
No, they are loca... well, wait, they're not. Sorry, that was an oversight from when I was testing them. They're local now. *looks around shiftily*

I don't like using module() for two reasons. One, if I don't use package.seeall, I have to save every outside function I want as a local before calling module(). Two, if I do use package.seeall, all of _G is put into the package table. There's a lengthy critique of module() on the Lua-Users wiki, and more often than not I find that I can manage well enough on my own. (You ask why not use module(), I say why use module()? Two different standpoints ;) )


Nick Gammon said:
Don't panic, I'll wait until all this is stabilized.

Thanks!

Nick Gammon said:

Twisol said:

With Load, I also return nil followed by an error message, though my messages are shorter and more useful to the calling script rather than the user (which makes sense, because the plugin will want to give its own error message).


Not in the version posted earlier on this page.

Ah, well, it's in my working revision. *cough*

[1] http://lua-users.org/wiki/LuaModuleFunctionCritiqued
Amended on Sat 09 Jan 2010 08:53 PM by Twisol
Australia Forum Administrator #27
Twisol said:

Two, if I do use package.seeall, all of _G is put into the package table.


No it isn't, "all of _G" is not put anywhere. All it does is put an __index entry in the package, pointing to the _G table. So, it *finds* the global variables, but can't change them. This is actually safer, you don't accidentally corrupt something in _G (like a counter or something). Here is the relevant code in Lua:


static int ll_seeall (lua_State *L) {
  luaL_checktype(L, 1, LUA_TTABLE);
  if (!lua_getmetatable(L, 1)) {
    lua_createtable(L, 0, 1); /* create new metatable */
    lua_pushvalue(L, -1);
    lua_setmetatable(L, 1);
  }
  lua_pushvalue(L, LUA_GLOBALSINDEX);
  lua_setfield(L, -2, "__index");  /* mt.__index = _G */
  return 0;
}


Admittedly, there is an overhead of the indirect lookup via the metatable, but then you are doing the exact same thing yourself in the module (using __index) so it can't be *too* bad. :P

If you were worried, for speed purposes, you would indeed copy all the relevant global functions into local variables. This would be faster again than your current approach, because using global variables (eg. CallPlugin, SetVariable) involves looking them up in the _G table every time.
USA #28
I meant that it was accessible through the package table, my apologies for my loose wording.

local myfile = [[
  module("test", package.seeall)
  function MYFOO()
    print("Hi.")
  end
]]

loadstring(myfile)()


tprint(test)
"MYFOO"=function: 03083340
"_M":
  "MYFOO"=function: 03083340
  "_M"=table: 041A55A8
  "_NAME"="test"
  "_PACKAGE"=""
"_NAME"="test"
"_PACKAGE"=""


tprint(getmetatable(test))
...
everything
...
"package":
    "preload":
    "loadlib"=function: 02F7A858
    "loaded":
      all other packages
      ...
      including this package
...



It's not so much the overhead of the lookups as it is the weirdness of it. Suffice to say that I like to have control over what my module does, and how.
Australia Forum Administrator #29
Well, back to exposing each function individually. I thought, "why not let Lua do it for us?". So a quick bit of code later, and you can find all the functions that need to be made local, like this:


local script = [====[
--[[

<< put your function code here >>

]====]

local found = {}
local libs_used = {}

local funcs = {}
local libraries = {
      "string",
      "package",
      "os",
      "io",
      "bc",
      "progress",
      "bit",
      "rex",
      "utils",
      "table",
      "math",
      "debug",
      "coroutine",
      "lpeg",
      "sqlite3",
      "world",
  } -- end of libraries

local tables = {
      "trigger_flag",
      "alias_flag",
      "timer_flag",
      "custom_colour",
      "error_code",
      "sendto",
   }  -- end of tables

for k, v in pairs (_G) do
  if type (v) == "function" then
    funcs [k] = true
  end -- if 
end -- for

for _, lib in ipairs (libraries) do
  for k, v in pairs (_G [lib]) do
    if type (v) == "function" then
      funcs [lib .. "." .. k] = true
      if lib == "world" then
        funcs [k] = true
      end -- if
    end -- if 
  end -- for

end -- for each library

-- now do stuff like error_code, colours etc.

for _, lib in ipairs (tables) do
  for k, v in pairs (_G [lib]) do
      funcs [lib .. "." .. k] = true
  end -- for
end -- for each table

-- scan our script for matching functions

for w in string.gmatch (script, "[%a%d._]+") do
  if funcs [w] then
    local lib, func = string.match (w, "^([%a%d_]+)%.([%a%d_]+)")
    if lib then
      libs_used [lib] = libs_used [lib] or {}
      libs_used [lib] [func] = true
    else
      found [w] = true
    end -- if
  end -- if  

end -- for

local tl = {}
for k in pairs (libs_used) do
  table.insert (tl, k)
end -- for

table.sort (tl)

for _, lib in ipairs (tl) do
  print ("local", lib, "= {")
  for k in pairs (libs_used [lib]) do
    print ("  ", k, "=", lib .. "." .. k .. ",")
  end -- for
  print ("  }")
end -- for

local t = {}
for k in pairs (found) do
  table.insert (t, k)
end -- for

table.insert (t, "_G")

table.sort (t)

print ("local", table.concat (t, ", "), "=", table.concat (t, ", "))


Now the output from that in the case of my earlier PPI module was this:


local error_code = {
eOK = error_code.eOK,
}
local package = {
seeall = package.seeall,
}
local string = {
format = string.format,
}
local CallPlugin, ColourNote, GetPluginInfo, GetPluginName, GetPluginVariable, IsPluginInstalled, Note, SetVariable, _G, assert, check, getfenv, load, loadstring, module, print, require, setfenv, setmetatable, type, unpack = CallPlugin, ColourNote, GetPluginInfo, GetPluginName, GetPluginVariable, IsPluginInstalled, Note, SetVariable, _G, assert, check, getfenv, load, loadstring, module, print, require, setfenv, setmetatable, type, unpack


Now you follow that by:


-- hide all except non-local variables
module (...)


And you are all set! It isn't perfect, in particular I assumed my coding style where I don't put a space between things like: string . format

Also, it doesn't notice comments, so if a function appeared only in a comment, it would still be entered in the list.


However it is an interesting illustration of using Lua to parse some Lua source and pull out useful things.
Amended on Sat 09 Jan 2010 10:38 PM by Nick Gammon
Australia Forum Administrator #30
By the way, there is a potential for a subtle problem. People *can* do this, although they are probably ill-advised to:


_G = nil


Now everything still works fine ... except anything that explicitly refers to _G (as yours and my code does). You can work around it like this:


local _G = getfenv ()

-- hide all except non-local variables
module (..., package.seeall)


Now we have a local copy of _G which is getfenv () (which is what _G really is anyway). Now it doesn't matter if the caller has mucked around with _G or not.

I must admit that some of the issues raised by the Lua group are valid (although they can nitpick at times).
USA #31
After a few days hard work, I finally implemented function-passing/returning, and Exporting of non-function values. For all intents and purposes, I think this is functionally complete. Linked below is the paste for PPI v1.2.0. [1]

Nick, I'd appreciate it if you could comment out the two lines on 307 and 308 to test the nonce-checking in version 4.46. It's supposed to store the nonce when the PPI is reloaded, and if it's changed, all thunks that refer to the other plugin will fail to execute (because there's no guarantee the same IDs refer to the same functions). So if the nonce -hasn't- changed, then reloaded should be false (the two commented-out lines), so the plugin knows not to send initialization stuff again.

Some testing of the new features with the below service seem to work very well:

PPI.Expose("n_test", 42)
PPI.Expose("s_test", "blue")
PPI.Expose("b_test", true)
PPI.Expose("z_test", nil)
PPI.Expose("t_test", {
  "test",
  42,
  true,
  nil,
  function()
    print("test from ATCP")
    return {true, 42}
  end,
})


Expected use of the nonce-checking in client code:

OnPluginListChanged = function()
  local atcp, reloaded = PPI.Load("7c08e2961c5e20e5bdbf7fc5")
  if not atcp then
    error("ATCP plugin not installed. Please install the ATCP plugin, then reinstall this plugin.")
  elseif reloaded then
    atcp.EnableModule("room_brief")
    _G["atcp"] = atcp
  end
end


[1] http://mushclient.pastebin.com/f1964fd71 see below post.
Amended on Mon 11 Jan 2010 10:10 PM by Twisol
USA #32
Just added a one-liner bugfix, as I realized that deserialize() would keep returning the outdated thunk if the other plugin had been reinstalled. Just added 'tbl.thunks = {}' to the Load function under tbl.nonce.

Also, being an idiot, the link I pasted in the above post goes to the wrong paste. I've removed that paste because I didn't even intend to paste it at all.


EDIT: Also, I'd like to point out that running an admittedly lightweight method across PPI 10,000 times takes only ~6.15 seconds. That's a paltry ~0.615 miliseconds (less than one ms) per call on average.

Granted, the same loop on the equivalent implementation within the same plugin space takes only 11 miliseconds to execute all 10,000 calls, but I don't think anyone really expected PPI to be quite that speedy.

local tstart = GetInfo(232)
for i=1,10000 do
  atcp.EnableModule('room_brief')
end

print(GetInfo(232) - tstart)



EDIT 2: By caching the indexing call as below, it cuts execution time using the PPI down to ~2.86 seconds for 10,000 calls, because it doesn't have to do a costly lookup every time. This follows the same logic as setting globals to locals before intensive loops in normal Lua, so it's not terribly unexpected either.

local tstart = GetInfo(232)
local EnableModule = atcp.EnableModule
for i=1,10000 do
  EnableModule('room_brief')
end

print(GetInfo(232) - tstart)


[1] v1.2.1: http://mushclient.pastebin.com/f6d7cd307
Amended on Mon 11 Jan 2010 10:30 PM by Twisol
Australia Forum Administrator #33
Just out of interest on the timing, and I agree it probably doesn't matter unless you are calling functions *a lot*, my earlier version is about 4 times as fast. Probably because it is setting up less variables than yours.

Doing 2 tests in succession, from your version I got:

 
Time taken = 4.364 seconds  (test 1)
Time taken = 5.216 seconds  (test 2)


From my version, calling the same functions:


Time taken = 1.137 seconds  (test 1)
Time taken = 1.168 seconds  (test 2)


My test code sent 2 arguments, and got 2 results:


-- client
for i = 1, 10000 do
  my_service.SomeMethodHere (42, "Nick")
end -- for


-- service
function SomeMethodHere (...)
  return 1, 2
end

USA #34
It could also be that I'm doing extra CallPlugin()s and serializations for the PPI_ACCESS message, because I support more than just methods. I would imagine that would slow it down a bit. Caching the index result as a local and calling that sped my test up to ~2.86 seconds on average, which is much closer to your total. Your version also doesn't support nested tables, which I imagine would also speed things up.

EDIT: I forgot to rename PPI_SUPPORTS to PPI_ACCESS in the message IDs section at the top. Well, re-pasted and version number updated (1.2.2)...

[1] PPI v1.2.2: http://mushclient.pastebin.com/f2cbbca69
Amended on Tue 12 Jan 2010 01:04 AM by Twisol
Australia Forum Administrator #35
Twisol said:

Your version also doesn't support nested tables, which I imagine would also speed things up.


Mine supports nested tables, just not tables that are self-referential or refer to other tables. For simple data passing (like passing a table of styles) that should be fine.
USA #36
Aren't tables that refer to other tables the same as tables within tables? I'm confused.

Testing with 'print(serialize.save_simple({1, 2, 3, {4, 5}}))' does indeed work, so I guess I just don't understand what you meant. Though, the fact that I support self-referential tables via a cache might add overhead.

I do notice that if save_simple comes across the same table twice, it treats them as separate tables, so on deserialization they will be separate objects. That's a problem my implementation does solve.

EDIT: Interesting, the non-_simple version of save does support self-referential tables, and tables that appear multiple times. Its output reminds me of an SQL dump. But ignore me, I'm just experimenting here.
Amended on Tue 12 Jan 2010 01:40 AM by Twisol
Australia Forum Administrator #37
Referring to other tables:


t1 = { 1, 2, 3, 4, 5 }
t2 = { 6, 7, t1 }


In that example, t2 refers to another table (t1). Thus t2 can't be simply serialized because you also need to serialize t1.

And this is self-reference:


t1 = {}
t1 [1] = t1


So t1, index 1, contains a reference to itself.

However tables within tables are OK:


t1 = { 
  wolf = { hp = 42, mana = 0 },
  naga = { hp = 100, mana = 10 }
  }


That is a simple nested table.
USA #38
Well, this:

t1 = { 
  wolf = { hp = 42, mana = 0 },
  naga = { hp = 100, mana = 10 }
  }


is precisely the same internally as this:

twolf = {hp = 42, mana = 0}
tnaga = {hp = 100, mana = 10}
t1 = {wolf = twolf, naga = tnaga}


Tables are like pointers, and other values are like normal values. It doesn't matter how you place the table pointers, because in the end they turn out the same.


Self-reference is of course the same internally, but we both know how differently it needs to be handled, since you can go into an infinite loop.


EDIT: Using serialize.save_simple(t1) on the second code snippet above gives the same output, i.e. this:

{
  naga = {
    hp = 100,
    mana = 10,
    },
  wolf = {
    hp = 42,
    mana = 0,
    },
  }
Amended on Tue 12 Jan 2010 02:26 AM by Twisol
Australia Forum Administrator #39
Twisol said:

Well, this:

t1 = { 
  wolf = { hp = 42, mana = 0 },
  naga = { hp = 100, mana = 10 }
  }


is precisely the same internally as this:

twolf = {hp = 42, mana = 0}
tnaga = {hp = 100, mana = 10}
t1 = {wolf = twolf, naga = tnaga}


Tables are like pointers, and other values are like normal values. It doesn't matter how you place the table pointers, because in the end they turn out the same.


Well, not precisely.

Your second snippet puts two tables into global namespace, mine doesn't. I know tables are like pointers, but your second example makes two additional tables globally available. It makes a difference.

eg.


_G ["twolf"] = {hp = 42, mana = 0}
_G ["tnaga"] = {hp = 100, mana = 10}
t1 = {wolf = twolf, naga = tnaga}



See? Your example make additional variables available to be accessed or modified without reference to t1. In my example, t1 "owns" the other tables.

Amended on Tue 12 Jan 2010 05:34 AM by Nick Gammon
USA #40
Well, sure. The point was that there's no difference in the tables themselves after all's said and done. I could have been pedantic and done this:

local twolf = {hp = 42, mana = 0}
local tnaga = {hp = 100, mana = 10}
t1 = {wolf = twolf, naga = tnaga}
twolf = nil
tnaga = nil


If it is said that a table contains another table, it's considered to be exactly the same as a table with a value that refers to another table, regardless of what's going on with external pointers to either of the two tables.

EDIT in response to:
Nick Gammon said:
Your second snippet puts two tables into global namespace, mine doesn't. I know tables are like pointers, but your second example makes two additional tables globally available. It makes a difference.

They're not additional tables if they exist regardless. The only extra 'things' involved are references to said tables.


I do find it a little funny (in a good way) that we're debating the exact interpretation of the composition of tables in this thread. *laughs*
Amended on Tue 12 Jan 2010 05:40 AM by Twisol
Australia Forum Administrator #41
Twisol said:

They're not additional tables if they exist regardless. The only extra 'things' involved are references to said tables.


My wording was "your second example makes two additional tables globally available". That's what it does. It adds two tables to global namespace. I don't debate that the two tables exist. I am saying that the difference is you have constructed three things in global namespace (the _G table if you like), whereas the earlier example constructed one thing in global namespace.

Your amended example using "local" has no extra impact on global namespace, granted, but I was working from your first example.
Australia Forum Administrator #42
I think the serialize.save_simple will serialize both examples correctly, however the serialize.save will correctly handle the case where both of the "extra" tables happen to be the same table.
Australia Forum Administrator #43
If I may say, I think that a system that handles serializing parameters involving tables that refer to themselves, or refer to other tables outside the main table, are unlikely to be required in any sort of reasonable API that plugins that might expose.

You may be solving a problem that doesn't exist, and indeed, that we should not encourage to come into existence.
Australia Forum Administrator #44
Twisol said:

I do find it a little funny (in a good way) that we're debating the exact interpretation of the composition of tables in this thread. *laughs*


Indeed. Lua is incredibly powerful, and it is good to understand the underlying mechanisms, particularly if you are involved in generic parameter-passing.

Thankfully, most of the time it is very simple to use.
USA #45
The self-referential serialization came for free by adding the serialization cache to properly cope with tables that appear multiple times. See:

t2 = {1, 2, 3}
t1 = {t2, t2}


Granted, this is extremely contrived, but your serialization routine (at least save_simple) treats both t2 appearances as separate tables, resulting in this:

{
  [1] = {
    [1] = 1,
    [2] = 2,
    [3] = 3,
    },
  [2] = {
    [1] = 1,
    [2] = 2,
    [3] = 3,
    },
  }


That's emphatically something I want to avoid. Like I said, self-referential came for free.
USA #46
To clarify, I've been arguing against this in particular:

Quote:
Referring to other tables:

t1 = { 1, 2, 3, 4, 5 }
t2 = { 6, 7, t1 }


In that example, t2 refers to another table (t1). Thus t2 can't be simply serialized because you also need to serialize t1.

[...]

However tables within tables are OK:

t1 = { 
  wolf = { hp = 42, mana = 0 },
  naga = { hp = 100, mana = 10 }
  }



EDIT: And just a couple posts ago, you said:
Nick Gammon said:
[...]or refer to other tables outside the main table, are unlikely to be required in any sort of reasonable API that plugins that might expose.


This is what I'm trying to drive at: there's no difference between a table 'referring' to a separate table, and a table 'containing' a separate table.
Amended on Tue 12 Jan 2010 06:00 AM by Twisol
USA #47
Nick Gammon said:
You may be solving a problem that doesn't exist, and indeed, that we should not encourage to come into existence.

This is an interesting point and one that should be given consideration.

The goal in these endeavors is not always to produce something that always behaves fully theoretically correctly, but that provides a good working environment for solving problems. This business of handling self-referencing may or may not be "too much", but in general I think it's worth thinking that designing an API/protocol involves making policy decisions, and not just solving technical problems. Solving some technical problem can end up being unhelpful in the end if it encourages (or even allows) confusing behavior that is not helpful to the overall policy.
USA #48
My goal is/was to create an inter-plugin communications system that behaves as closely as possible to the real thing. With that in mind, I wanted to make sure that table references were not duplicated but preserved, so the end serialization would not be 'lossy', so to speak. Serializing self-referential tables, or even just tables that somehow happen to have themselves as a descendent somewhere along the line, came for free due to the implementation of the cache.

I also try very hard not to make assumptions about what the end user will do with it, but stick to my goal with the software instead. That is, if the end-user has some reason to want to pass a self-referential table, go for it. Theoretically, you might have a plugin that handles on-close serialization for other plugins, so it shouldn't care what the format of the tables are. Its job is just to do something with whatever it was sent. I don't think that's confusing.

There are two things I did decide not to implement: metatables and writing to the PPI. It's rather complex and, yes, not suitable for a protocol that's only really meant to be communicating data back and forth. (By the by, passing methods across is sort of important, as it allows both plugins to communicate in more interesting ways depending on the API of the plugins involved.)


EDIT: Just to make sure you understand, that on-close serialization plugin is probably not a very good idea, mainly because it would have to re-serialize the arguments it's passed. I just brought it up as a theoretical example of a plugin that doesn't require any specific data format.
Amended on Tue 12 Jan 2010 06:22 PM by Twisol
USA #49
Twisol said:
I also try very hard not to make assumptions about what the end user will do with it, but stick to my goal with the software instead.

As a general comment, the best APIs are the ones that do in fact make (extremely judicious) assumptions about what the user will be doing.

I haven't really thought this very particular issue through, but complexity for the sake of technical correctness is not necessarily a desirable outcome. It very well might be a good thing to allow self-referential tables, but then again maybe those tables are not the kind of thing that would/should be communicated, and allowing the communication would allow things to be passed around that probably shouldn't (e.g., _G).

In this case, it's not really that passing self-referential tables makes the library hard to use, it's that it allows behavior that one might not want to allow from a policy perspective. This is a very different question from the technical question of how one does it. Perhaps the distinction is subtle.

I don't mean, and I don't think Nick meant, this as a comment on this particular library or even this very particular point, but rather as a comment on general API design borne from experience in seeing too many "do it all" libraries. "Complexity" here does not mean "difficulty of understanding", but rather extra functionality that may be nice but not directly relevant to the problem being solved. Perfection being found in the end of removals, and all that.
USA #50
David Haley said:
As a general comment, the best APIs are the ones that do in fact make (extremely judicious) assumptions about what the user will be doing.

Let's say that my assumption is that they want to transport tables that contain data, which may include other tables. A doubly-linked list sounds like something I'd want to allow, for example.

David Haley said:
[...] would allow things to be passed around that probably shouldn't (e.g., _G).

All I can say: blatant user error. There are plenty of things you shouldn't do with libraries, yet you can. Passing _G would be one of those very stupid things. Using (.*) in PCRE is very slow when you have to backtrack a lot. Using #include to include a .c file is generally not a good idea, unless it happens to deal with C++ templates (in which case it's one of the only easy ways to separate the declaration and definition into two files, IIRC). You -can- do these things, but in general you absolutely should not. Yet there are some cases where you need to. (There's never any excuse for passing _G, but my point is that it's almost impossible to stop it)


EDIT: To be pedantic, impossible to stop it without, say, making it hard/impossible to pass linked lists. Someone could carefully use setfenv() on the PPI to give it a fake _G, and then pass the real one, so comparisons would be futile.
Amended on Tue 12 Jan 2010 06:54 PM by Twisol
USA #51
Quote:
A doubly-linked list sounds like something I'd want to allow, for example.

Really? Shouldn't you be passing the data, not the implementation? In some languages, a straightforward list would be a vastly superior solution to a contraption from another language implementing a doubly-linked list. Even in Lua, a doubly-linked list is not very natural to work with; you would do so using all kinds of metatables. As a Lua programmer, getting a doubly-linked list instead of a plain old array would be rather annoying unless there's an awfully good reason for the list to be doubly-linked.

This is actually a pretty good example of something you might want to make as policy: data should be passed by its semantics, not by any implementation details.



That said, circular references are something that do come up. It's unclear to me if you actually want to communicate these circular references, as again they are usually implementation details. But, simple examples are tree or other container structures where children refer to their parents.

This is still of course communicating implementation rather than data. A preferable way of communicating a tree across language boundaries, IMO, would be using an API for constructing trees and passing it the parameters. This lets every language choose the most appropriate data structure for trees. (Perhaps the data structure used in one language is absolutely lousy in another compared to a built-in.)
USA #52
For posterity, a cleaned-up portion of a conversation David and I had on the IMC network:

MudBytes IMC said:
[ichat] DavidHaley@MW: Perhaps the problem is that the "context" of PPI isn't necessarily clear

[ichat] Twisol@Talon: Improving CallPlugin.

[ichat] DavidHaley@MW: OK. But does that mean transporting low-level data structures, or high-level semantics?

[ichat] Twisol@Talon: PPI is supposed to implement as sensible as possible of an interface for the user in whatever language it's ported to.

[ichat] Twisol@Talon: I don't really get the difference.

[ichat] DavidHaley@MW: Do you know Java?

[ichat] Twisol@Talon: Decently well.

[ichat] DavidHaley@MW: OK. So it's like the difference of communicating a List (high-level) vs. an ArrayList or LinkedList (implementation details)

[ichat] Twisol@Talon: I actually had the thought that PPI was as similar a protocol to Java's RNI than I could ever hope to write.

[ichat] Twisol@Talon: The problem is that the data needs to be sent across somehow, yes?

[ichat] DavidHaley@MW: of course

[ichat] Twisol@Talon: you can't think of it in terms of classes/interfaces. *shrug*

[ichat] DavidHaley@MW: Why not?

[ichat] Twisol@Talon: the data is data, and treatign it as a generic List requires morphing the dataset somehow to match your idea of a generic list

[ichat] DavidHaley@MW: I don't mean a literal class in the scripting language, by the way, but a concept that the protocol knows how to deal with.

[ichat] Twisol@Talon: Again, write your own protocol over PPI.

[ichat] Twisol@Talon: Hmm, lightbulb.

[ichat] Twisol@Talon: PPI is the lower-level transport protocol.

[ichat] DavidHaley@MW: Well, that's a good way of defining it

[ichat] Twisol@Talon: An idea that comes to mind is something akin to TCP/IP or UDP/IP. Build a protocol called WTF over PPI, and it would be WTF/PPI.

[ichat] DavidHaley@MW: That mission statement wasn't really clear earlier, IMO

[ichat] Twisol@Talon: No, you're right.

[ichat] Twisol@Talon: I knew what I wanted, I just didn't say it right.

[ichat] DavidHaley@MW: welcome to the world of why API design is so hard :D

[ichat] Twisol@Talon: whatever language the stuff reaches, it can build its own representation of it based on the format received

[ichat] Twisol@Talon: *nod*

[ichat] DavidHaley@MW: yes, it seems that way (again I'd have to poke at it a lot more to make a more definitive statement, but at first glance it seems pretty reasonable)
Australia Forum Administrator #53
I'm still not sure what problem the more complex PPI module solves, that needs to be solved, other than as an academic exercise. You should take a look at the Lua forums one day, there are many posts there about adding obscure features to Lua to solve obscure problems, many of which the authors are resisting.

Just as an example, maybe a metatable to detect data changing in a table, rather than a missing index or adding a new index.

However, every enhancement comes at a cost, and in the case of this extra metatable entry, which you could probably argue would be useful, would add an overhead to *every* Lua program, including ones which don't require the feature.

Twisol, you argued in an earlier page in this thread that you wanted a more language-neutral solution (than the Lua serialized table) for passing parameters, but later on you talk about "not serializing metatables". Well of course, metatables are specific to Lua, so even if you attempted to do so, you would then have a Lua-specific solution again.

Even tables, self-referential or not, may not exist in every language. From memory, VBscript actually only has arrays, not tables indexed by arbitrary keys.

Thus a truly language-neutral API would keep to very simple arguments (eg. numbers, strings, booleans) and possibly simple single-level tables indexed by a numeric index.

Even with my method of passing parameters, if you had to set them up in a different language (like, Python) it wouldn't be hard to fudge up something that looked exactly right to match the serialize.save_simple output. OK, we need a number, a string, and a boolean. So it would be:


{
  [1] = 42,
  [2] = "nick",
  [3] = true,
  }


It would be easy for any language to produce something like that (basically you have the argument number in brackets, an "=" sign, and the data after it).

The type of data is implied. If it is quoted, it is a string. If it is a number, it is numeric. The words "true", "false", and "nil" have specific meanings. And if the value starts with a "{" then you have a sub-table which is easy to set up as well.
USA #54
a) PPI is an optional library, not a core feature of Lua that you have whether you like it or not.

b) I kept in mind the (lack of) functionality of other languages in designing the serialized data protocol. Keys can only be numbers and strings, values can be numbers, strings, booleans, methods, nil, or other tables/lists. The ports of PPI have a few choices when it comes to dealing with data it can't represent exactly how it came in: in the case of keys, defaulting to string should be acceptable. Or it could implement its own custom data model if possible, like David suggested: an API for explicitly adding items to be sent.

c) Methods are simply passed around by ID, so I believe any decent scripting language could write a shim like my new_thunk that eval()s and returns a thunk.

d) My serialization isn't actually that complex, either. Every MUSH variable contains a single 'table' or 'list' of entries. Your example table above would be:

n:1|n:42|n:2|s:nick|n:3|b:1


This is generated using ArrayCreate(), ArraySet(), and ArrayExport(), so the string manipulation isn't much of an issue either.

In the simple case, my serialization supports everything yours does. In the complex case, it supports a bit more, meaning languages that support those features can take advantage of them. In the simple case, every language could theoretically utilize the PPI, using z:~ to serialize values it can't serialize properly, and using nil/null/whatever to deserialize values it doesn't understand. More complex operations could necessitate the use of a more powerful language, but I don't see anything wrong with that. After all, if a language inherently doesn't allow you do to something you want to do...

e) You mentioned metatables and how I didn't choose to support them. The MUSHclient-supported languages that could support any form of metatables are very few AFAIK, and it would be hardly worth it at all when a proxy table could be used instead.


As an aside, you could emulate that data-changing metamethod using __newindex and simply writing to a separate table instead of the one operated on. Clearly, there's an easier alternative. No wonder the Lua developers don't care to add a new metamethod.
Amended on Tue 12 Jan 2010 08:06 PM by Twisol
USA #55
Twisol said:
a) PPI is an optional library, not a core feature of Lua that you have whether you like it or not.

I might have misunderstood, but isn't it being proposed as the standardized MUSHclient solution for communication between plugins?

It might be nice to frame the problem as precisely as possible. Having a method of communicating low-level information is useful. But it's also useful to have a method of communicating high-level information. The latter is (probably) a more common problem for scripting authors; I would posit that it's relatively rare to care about the data structures themselves.

This comment might be irrelevant depending on the precise problem being solved and how it will fit in to MUSHclient.

Twisol said:
As an aside, you could emulate that data-changing metamethod using __newindex and simply writing to a separate table instead of the one operated on. Clearly, there's an easier alternative. No wonder the Lua developers don't care to add a new metamethod.

It's not really so simple. The proxy table approach presents many complications of its own. (For starters, the public interface is no longer a table with keys and values, but an empty table with a metatable. Serializing such a thing is more difficult.) There are good reasons why one might want a data-change event, just as there are good reasons why one might not want it.
EDIT: for example, a very real problem with proxy tables is determining the set of keys that a proxy table has values for. This is very, very annoying to work around. IIRC, the Lua authors are adding a __keys metamethod to Lua 5.2 for this very reason.
Amended on Tue 12 Jan 2010 08:14 PM by David Haley
USA #56
David Haley said:

Twisol said:
a) PPI is an optional library, not a core feature of Lua that you have whether you like it or not.

I might have misunderstood, but isn't it being proposed as the standardized MUSHclient solution for communication between plugins?

I meant optional as in, if a plugin doesn't want to communicate with others, it doesn't have to include the library.

David Haley said:

It might be nice to frame the problem as precisely as possible. Having a method of communicating low-level information is useful. But it's also useful to have a method of communicating high-level information. The latter is (probably) a more common problem for scripting authors; I would posit that it's relatively rare to care about the data structures themselves.

This comment might be irrelevant depending on the precise problem being solved and how it will fit in to MUSHclient.

PPI is just a transport library. How the data is interpreted will be defined by the service exposing the methods. If a plugin wants to communicate with the service, it will need to follow the 'contract', so to speak, of the service's interface.

David Haley said:

Twisol said:
As an aside, you could emulate that data-changing metamethod using __newindex and simply writing to a separate table instead of the one operated on. Clearly, there's an easier alternative. No wonder the Lua developers don't care to add a new metamethod.

It's not really so simple. The proxy table approach presents many complications of its own. (For starters, the public interface is no longer a table with keys and values, but an empty table with a metatable. Serializing such a thing is more difficult.) There are good reasons why one might want a data-change event, just as there are good reasons why one might not want it.
EDIT: for example, a very real problem with proxy tables is determining the set of keys that a proxy table has values for. This is very, very annoying to work around. IIRC, the Lua authors are adding a __keys metamethod to Lua 5.2 for this very reason.

I'll give you that one. __keys does sound like it would solve those problems quite nicely.
USA #57
At any rate, if everyone really believes PPI needs to be trimmed down, despite all of my reservations, I can do it. What in particular do you think should be changed, and what benefits will it bring to not include it?
USA #58
It's not that it needs to be trimmed down per se. It's that we need to (a) very clearly define the problem it's solving, (b) identify the problems that people need solved in practice, and (c) figure out how we can best map those two to each other.

The solution adopted in the end of the day (to the problem that we will eventually identify) might very well end up being something else that uses PPI, it's hard for me to tell at this early stage.

This isn't meant to sound patronizing in the slightest -- I hope it doesn't :( -- but one thing I would tell my students is to step away from implementation, and think about the higher-level problem being solved -- as it relates to their practical problem. Obviously we have a problem of getting data between plugins. But that problem statement is very general! It can mean anything from throwing around numbers and strings, to arbitrarily complex data structures, and maybe even a full RPC mechanism, heck why not with objects and classes too. What are the requirements being addressed? If we have callbacks, to what extent does one need to invoke them? Etc.
In other words: what are the use cases being targeted? (This is a different question from: What can we do with this?)
USA #59
My original goal was to replace LoadPPI with something decent, safe, and useful. The original use-case was calling to a service plugin (such as ATCP) to tell it something or have it do something. Even with LoadPPI though, the service plugin had to write CallPlugin() calls and come up with some way to pass the parameters onwards. PPI brings that into itself, freeing the service from having to deal with those low-level details.

Asynchronous callbacks, which you mentioned, are something I also want to use in my ATCP plugin. You simply register for a certain message, and when it arrives, it calls directly to your callback. Callbacks are just an extension of the general PPI idea, except instead of calling a method pulled from the loaded PPI interface, you're calling a separate thunk. It's the same internal workings, though.

An alternative is to scan all plugins for an arbitrary OnPluginAtcp function and call that directly, but that would require the client to also implement a service for the purposes of using OnPluginAtcp.
Amended on Tue 12 Jan 2010 08:53 PM by Twisol
USA #60
While converting my ATCP plugin to use PPI, I came across two issues: disabled service plugins (as opposed to not-installed plugins) weren't recognized in Load and caused PPI to break, and if an error occured in the service method, stuff would break in the client too. The former was rectified with an extra check in Load, and the latter with an even more minor modification in deserialize(). The updated version is linked below.

http://mushclient.pastebin.com/f16a36fbc
USA #61
So what's our status here? It seemed to kind of fall off pretty quickly. I'd like to see my full-featured version included in 4.46, obviously - especially because my ATCP plugin now uses function callbacks to great effect, and I'd be loath to give that up - but I'm still very willing to change things if you think something's out of place.

I also added a small bit to _G[request_msg] to pass the calling plugin's ID as the first parameter, because I can see very real instances when you'd want to know the caller's ID.

  -- Call method, get return values
  local returns = {func(id, unpack(params))}
  
  -- Send returns
  send_params(returns)
end
USA #62
As a side-note, I remembered us discussing module() earlier in the thread, and I came up with this function to create a new ready-to-use environment containing only what would have come with the environment to begin with. It's not perfect, of course - it shares the same subtables with the original _G. But it does do a good job of creating a standalone environment that a library can use without worrying about locals.

function basefenv()
  local _G = getfenv(1)
  
  local _G2 = {
    __VERSION = _G.__VERSION,
    assert = _G.assert,
    collectgarbage = _G.collectgarbage,
    dofile = _G.dofile,
    error = _G.error,
    getfenv = _G.getfenv,
    getmetatable = _G.getmetatable,
    ipairs = _G.ipairs,
    load = _G.load,
    loadfile = _G.loadfile,
    loadstring = _G.loadstring,
    module = _G.module,
    next = _G.next,
    pairs = _G.pairs,
    pcall = _G.pcall,
    print = _G.print,
    rawequal = _G.rawequal,
    rawget = _G.rawget,
    rawset = _G.rawset,
    require = _G.require,
    select = _G.select,
    setfenv = _G.setfenv,
    setmetatable = _G.setmetatable,
    tonumber = _G.tonumber,
    tostring = _G.tostring,
    type = _G.type,
    unpack = _G.unpack,
    xpcall = _G.xpcall,
    
    bc = _G.bc,
    bit = _G.bit,
    coroutine = _G.coroutine,
    debug = _G.debug,
    io = _G.io,
    lpeg = _G.lpeg,
    math = _G.math,
    os = _G.os,
    package = _G.package,
    progress = _G.progress,
    rex = _G.rex,
    sqlite3 = _G.sqlite3,
    string = _G.string,
    table = _G.table,
    utils = _G.utils,
    world = _G.world,
    
    check = _G.check,
    gcinfo = _G.gcinfo,
    loadlib = _G.loadlib,
    
    alias_flag = _G.alias_flag,
    colour_names = _G.colour_names,
    custom_colour = _G.custom_colour,
    error_code = _G.error_code,
    error_desc = _G.error_desc,
    extended_colours = _G.extended_colours,
    sendto = _G.sendto,
    timer_flag = _G.timer_flag,
    trigger_flag = _G.trigger_flag
  }
  _G2._G = _G2
  
  return setmetatable(_G2, getmetatable(_G))
end
Australia Forum Administrator #63
Twisol said:

So what's our status here? It seemed to kind of fall off pretty quickly.


Well I was waiting to see what happened, and if anyone else would comment.

My attitude towards incorporating verbatim your code in the next release can be summed up somewhat by the comments made by the Lua developers about accepting patches, on this page:

http://www.lua.org/faq.html#1.9

Lua developers said:

1.9 Do you accept patches?

We encourage discussions based on tested code solutions for problems and enhancements, but we never incorporate third-party code verbatim. We always try to understand the issue and the proposed solution and then, if we choose to address the issue, we provide our own code.


I am very grateful to you for raising the issue of plugin-to-plugin communication. However I believe that my simplified version addresses the situation adequately. So far, out of the thousands of MUSHclient users, and the dozens of plugin developers who frequent these forum pages, there has been almost complete silence about whether this module addresses any needs that are currently considered requiring addressing.

Since I believe my version provides a subset of your version, an upgrade could be considered in future if it was needed (although your latest patch which supplied the ID of the calling module might break this).

As a general rule, if there are two solutions to a problem, I prefer the simpler one. In years to come the simpler one is generally easier to understand, and to fix or modify if needed.

In any case, as this module is not part of the core MUSHclient code (that is, the .exe file), if you required your more sophisticated version when distributing plugins, you can, of course, include it.
USA #64
To me, it's not about seeing a need and filling only the exact requirements of it, it's about providing options that might spark more ideas. Anyways, if I had known this was about getting voices, I'd have pointed this thread out to the plugin developers in Achaea who don't frequent these forums by any stretch of the imagination.

I would like to ask what you mean by "simple", though. Simple to the end user, or in the implementation?

I agree that I could still distribute my own version, but that would still require an extra download for the users of the plugins (not just the developers), which was the driving force behind my creating LoadPPI originally. Both of our versions can be used exactly the same way (minus the ID patch), with exactly the same semantics, except that mine is simply more flexible. I'm willing to take the hit in code 'complexity' (and my version isn't very complex, IMO) if it means the user has an easier time of it.


I know I'm beating a dead horse, I'm sorry about that. It's not really about the code, I just honestly don't understand... You're leaving out features that I think are really useful, and I've used them in a plugin (ATCP) that benefits greatly from said features (mainly function callbacks as parameters), for no other reason than simplicity in the module itself? Again, I'm not trying to flame you, troll you, or be a bad sport in general. I just don't understand, and that's what I'm trying to figure out.
Australia Forum Administrator #65
OK, well it would help if you documented what your version does exactly, and what the extra features are. A programming exercise is not really done until it is documented, otherwise people don't know what the code does, how to use it, and why they might want to.

My proposed version on page 2 of this thread has about 80 lines of documentation at the start (as a comment), yours has none.

In the middle, for example, your module has "Called to access and return a value from the service". What does this do? Is this on top of calling the function itself, but gets a value separately? Why do this? Can you explain in some detail all these extra features and what they are useful for?

  • Given that both yours and my module achieve approximately the same thing (simplify calling a function in another plugin, passing multiple arguments, and getting multiple values as a return), I note that my version, excluding the initial comments, is 90 lines of code whereas yours is 374 lines.
  • My tests showed my version worked about 4 times as fast as yours.
  • My version only uses a single MUSHclient variable for each call (for all the return values), whereas yours uses a variable per argument, and another variable for each return value.
  • You have rewritten serialization for reasons that are a bit obscure to me, considering there is already a "serialize" module. The existing serialize module has had the test of time, in terms of ironing out wrinkles.
  • You say you want to be able to serialize tables with self-references, or references to other tables, but neither David Haley nor I can see why you would want to do that, apart from showing it can be done. In any case, the existing serialize module will do that if you change from serialize.save_simple to just serialize.save.
  • Can some of the extra features, like finding what a plugin supports, not be layered on top of my simpler implementation, if required? And would they be required?
  • I find code like this just obscure:

    
     PPI_list[id] = tbl
     PPI_list[tbl.ppi] = id
    


    It appears to be storing A inside B, and then B inside A. I'm not sure why this is required.
  • You mention an important feature is "function callbacks as parameters". Can you explain that? Give an example? Are you passing a function from one plugin to another? Is this required?


I would like a bit more peer review of this code (including mine). In other words, an assessment by other plugin-writers, than you or me.

Every couple of days you have been posting bug fixes or minor improvements. One thing at least I would like to see if a week or two elapse where no further changes were made (obviously you could achieve this by simply not posting) but not only that, to have other people use it and say "yes it works as advertised". Of course, to work as advertised you need to do what I said at the start of this post, and document exactly what it does, giving examples of usage.

Amended on Mon 18 Jan 2010 09:24 PM by Nick Gammon
USA #66
Alright, this is more like it. I'll get back to you ASAP once I've finished. ;)
USA #67
Firstly, here's an in-depth look at the module itself. I hope it helps explain things for you. I'll go back over each point you raised in a separate post.


My PPI implementation communicates with other PPIs by means of three 'messages': ACCESS, REQUEST, and CLEANUP. REQUEST should arguably be INVOKE, but it's a remnant of the original implementation. These messages are sent through CallPlugin(), passing the client plugin ID as its single parameter.

ACCESS: Retreives a value from the service by name. The value can be any of: string, number, boolean, table, function.
REQUEST: Invokes a remote function by ID. Parameters are serialized into MUSHclient variables for access by the service. Return values are serialized by the service and retreived afterwards. Any of the following values can be passed or returned: string, number, boolean, table, function.
CLEANUP: Called after every ACCESS or REQUEST message to remove the temporary variables involved.

All variables are passed by value. Modifications to a local copy have no effect on the original value.

The serialization format for an individual value is simple enough:

Strings: "s:a string"
Numbers: "n:42"
Booleans: "b:1"
Tables: "t:1" *
Functions: "f:1" **
Nil, or unknown: "z:~"

* Tables are serialized separately, and each is given its own MUSHclient variable and an ID. It is very important to note all subtables within a table are serialized as well, so serializing _G would probably be a bad idea.

** Functions are not serialized, per se; they are stored locally and given a unique identifier, which is used in the serialized list of parameters. When deserialized, a new resolver function is created, which when invoked will send a REQUEST message on that method ID to the service.

These serialized values are inserted into a MUSHclient array and exported, resulting in a string like this:

n:1|s:a string|n:2|b:1

The keys are also serialized, as it is legal to serialize a table with either string or numeric keys. The above line could be seen more as:

n:1 | s:a string
n:2 | b:1



In order to communicate with another plugin in the first place, you must create a PPI proxy. This is done with PPI.Load(), using the ID of the service plugin as the only parameter:

test = PPI.Load("7f88d638ba84f0c48b646dd2")



A simple access and subsequent invocation looks like this:

local tbl_param = {4}
test.FooBar(true, "2", 3, tbl_param, tbl_param)


Breaking it down, we have this series of steps:

1. "test.FooBar" causes an ACCESS message to fire. It retrieves the value stored remotely as FooBar, which could be any of the acceptable types mentioned previously. In this case, we assume it's a function.

2. "FooBar(...)" causes a REQUEST message to fire through the retrieved method. The arguments are serialized together as a table. The resulting variables used would look like this:

PPIparams1: n:1|f:1|n:2|b:1|n:3|s:2|n:4|n:3|n:5|t:2|n:6|t:2
PPIparams2: n:1|n:4


As you can see, the entire original parameters list itself is serialized as a single variable. Extra variables are used as needed to serialize nested tables, such as tbl_param. As well, even if a subtable appears more than once, it is only ever serialized once.

The function ID being invoked is passed as the first parameter, which is why there are six items rather than five.


A service exposes values to the world by using PPI.Expose(). This method takes two parameters: the key by which clients will retreive your value, and the value to store.

_G.callbacks_list = {}

PPI.Expose("number", 42)
PPI.Expose("boolean", true)
PPI.Expose("string", "My Plugin")
PPI.Expose("table", {
  ["a key"] = "a value",
  [1] = 10,
  [2] = 50,
})
PPI.Expose("StoreCallback", 
  function(id, callback)
    table.insert(_G.callbacks_list, callback)
  end
)
PPI.Expose("ExecuteCallbacks",
  function(id, ...)
    for _,v in ipairs(_G.callbacks_list) do
      v(...)
    end
  end
)


This code exposes a series of values, covering every type supported by PPI. Here is an example client and its output.

test = PPI.Load("7f88d638ba84f0c48b646dd2")
require('tprint')

print("boolean: ", test.boolean)
print("string: ", test.string)
print("table:")
tprint(test.table)
print()
print("function: ", test.StoreCallback)

myfoo = 10

test.StoreCallback(function(id, ...) tprint{...} end)
test.StoreCallback(function(id, ...) print() print(myfoo, " - ", #{...}) end)
test.ExecuteCallbacks(10, 15, 30, "woohoo")


Output:
function: function: 01459910
boolean: true
string: My Plugin
table:
1=10
2=50
"a key"="a value"

function: function: 014598E0
1=10
2=15
3=30
4="woohoo"

10 - 4


And of course, if you try to access a value that's nonexistant, the client gets nil. The passing of functions (or more accurately function proxies) allows plugin developers to make use of the subscription model. Just like BroadcastPlugin() sends data to all plugins (interested or not), a service can expose a 'subscription' method for plugins to pass a callback to, and that plugin can later execute those callbacks it received.
Amended on Mon 18 Jan 2010 10:45 PM by Twisol
USA #68
I don't mind putting some tutorial documentation in the module, but I prefer to keep it separate from the file, such as online, for more comfortable reading.

Nick Gammon said:
Given that both yours and my module achieve approximately the same thing (simplify calling a function in another plugin, passing multiple arguments, and getting multiple values as a return), I note that my version, excluding the initial comments, is 90 lines of code whereas yours is 374 lines.

I personally don't think code size is a good comparison. Mine has more lines - many of which are whitespace and comments, by the way - mainly because I wrote my own serialization functions (in order to 'serialize' functions properly). A good design also tends to be 'larger' in general, but it's also easier to extend/maintain. (I'd like to think this is designed at least decently.)

Nick Gammon said:
My tests showed my version worked about 4 times as fast as yours.

Correction: maybe twice as fast when compared to the version caching the retrieved method. Your version sends only one CallPlugin message, with simpler serialization. Mine sends two (one access, one request/invoke), with slightly more complex serialization to accomodate PPI-specific requirements. In my first test, both access and request messages were sent every iteration. In the second, I cached the access message first in a local, then used that in the loop. That's standard practice in Lua anyways (we discussed this in relation to module() and storing methods in locals before calling it).

Nick Gammon said:
My version only uses a single MUSHclient variable for each call (for all the return values), whereas yours uses a variable per argument, and another variable for each return value.

Incorrect. Mine uses a variable per table. The parameters list is treated as a single table itself, so if you pass no tables as parameters, no extra variables are used. This applies identically to the return values. Yes, in a past version I used a variable for each argument/return, but that's no longer the case.

Nick Gammon said:
You have rewritten serialization for reasons that are a bit obscure to me, considering there is already a "serialize" module. The existing serialize module has had the test of time, in terms of ironing out wrinkles.

True. However, I wanted the PPI communications protocol to be as unhindered by language as possible, and I also wanted to be able to allow functions (or rather, function identifiers) to be passed. It turned out to be easiest to write my own, but if you see any flaws or bottlenecks in my implementation, please let me know.

Nick Gammon said:
You say you want to be able to serialize tables with self-references, or references to other tables, but neither David Haley nor I can see why you would want to do that, apart from showing it can be done. In any case, the existing serialize module will do that if you change from serialize.save_simple to just serialize.save.

In reference to the bolded... excuse me if I sound a bit tweaked, but there is no difference between a table containing a table, and a table containing a reference to another table. Clearly, we want to be able to pass tables containing tables, you have said as much yourself. I'm not understanding this.

In relation to the rest of the quote, yes, serialize.save will do the job properly and without subtable duplication, but I found it was easiest to roll my own when it came to the PPI communication format.

Nick Gammon said:
Can some of the extra features, like finding what a plugin supports, not be layered on top of my simpler implementation, if required? And would they be required?

I'm not sure what you mean by "finding what a plugin supports". The old SUPPORTS from before was adapted into the current ACCESS, which simply retrieves a value from the PPI, whether it be typical values or a method. Any kind of 'supports' checking is simply done by checking the value against nil.

Nick Gammon said:
*I find code like this just obscure:


 PPI_list[id] = tbl
 PPI_list[tbl.ppi] = id


It appears to be storing A inside B, and then B inside A. I'm not sure why this is required.

Incorrect, although I agree that it isn't clear. I'm storing two sets of data in the PPI_list table: one, a list of client PPI records (including private data) keyed by service ID; and two, the ID of the client PPI keyed by the actual PPI object contained within the record (the one that the user is given). It's confusing, but the items in the record ('tbl') are private data, and tbl.ppi is the PPI proxy the user uses. For the most part, the second item (the id keyed by the PPI) is used in the PPI metatable's __index in order to get at the private data. I could have used a separate table to store the ppi-to-id mappings, but it was easier just to do this, since the keys couldn't ever clash.

Your analogy would be more A.c = B; B.c = A. That's not the case, because here it's the same table being inserted into both times. It just happens to be odd keying.

Nick Gammon said:
*You mention an important feature is "function callbacks as parameters". Can you explain that? Give an example? Are you passing a function from one plugin to another? Is this required?

I think my above overview showed a good example of this. Yes, it's absolutely required in order to sanely implement selective broadcasting (aka event subscription). My ATCP plugin uses it: a client plugin just tells it what message it wants to listen for, and a callback to call when it comes in. It's very clean and clear.


EDIT: Rather belated fix from 'model' to 'module'.
Amended on Tue 19 Jan 2010 05:00 AM by Twisol
USA #69
Quote:
ACCESS: Retreives a value from the service by name. The value can be any of: string, number, boolean, table, function.
REQUEST: Invokes a remote function by ID. Parameters are serialized into MUSHclient variables for access by the service. Return values are serialized by the service and retreived afterwards. Any of the following values can be passed or returned: string, number, boolean, table, function.
CLEANUP: Called after every ACCESS or REQUEST message to remove the temporary variables involved.

An internal 'cleanup' might or might not be needed for implementation reasons, but these are precisely implementation details and the user should never, ever have to worry about cleaning up implementation-specific temporaries.

As soon as I 'access' data or 'request' a function's return value, I have the values in hand and they should go away immediately in any other form they might exist in other than the values I have.

Quote:
with slightly more complex serialization to accomodate PPI-specific requirements

It's still a little unclear to me, to be honest, if these requirements were driven by actual problems to solve, or problems that could be solved.

Quote:
but there is no difference between a table containing a table, and a table containing a reference to another table.

Actually, there sort of is. You can think of it as the difference between a containment tree and a containment graph. I think we all agree that we want to be able to pass tables with subtables; it's still unclear to me (and Nick, I believe) why we want tables that can refer to themselves (which is what arbitrary table references allow).

Quote:
but I found it was easiest to roll my own

A general rule of thumb is to avoid rolling your own unless you really, really have to.


For whatever it's worth, your serialization code (as posted on page 2, v1.0.1, was the most recent I saw in the thread) does not allow a table key to be anything but a string or number, whereas Nick's can (with some limitations). Before you point at the limitations, I would continue to argue that limitations are only a problem insofar as they prevent the tasks we actually have from being accomplished. To be fair, table keys of type 'table' are perhaps weird in the first place, but part of your argument is being language-independent and as general as possible. So this could be a source of confusion. In fact, your serialization function silently ignores stuff it can't serialize, which could lead to very confusing problems!



TBH, and I hate to harp on this, but it still seems to me like the requirements haven't really been defined. I'm not talking about implementation requirements, I'm talking about API requirements. We're talking of "PPI requirements" but I haven't seen a single place where the high-level goals are cleanly laid out, with real-world use cases and justifications for each of the requirements.

Complexity in general, solving problems simply because you can, is not always a good thing. This is a Very Important Concept and perhaps hard to explain succinctly. There's a good quote about perfection by Antoine de Saint-Exupery; paraphrasing, the designer knows that he has achieved perfection not when there is nothing left to add but when there is nothing left to take away. This is extraordinarily relevant to computer science in general and API/protocol/etc. design in particular. Solving all kinds of problems sounds awfully nice, until you try to solve so many that you end up confusing the user when it comes to solving the problems you initially had. This is a somewhat lengthy reference, but I would recommend this document:
http://cacm.acm.org/magazines/2009/5/24646-api-design-matters/fulltext
(Caveat: I've only read about half of it so far, myself)

To summarize: I find it very difficult to comment without having a very clear picture of the exact problem we're solving and why it's a problem. This should be general, but more specific than just "sending stuff between plugins" -- that much is obvious enough. ;-)



Here's an example: we need to send functions around. Do we really need to send functions that don't have proper names? If we only send properly named functions, then we can dramatically simplify the passing-around of functions; namely, you specify them by name and you're done. Yes, yes, it would be nice to construct arbitrary callbacks in Lua using closures etc. and then be able to pass these to VBscript or whatever. But this introduces complexity for the entire library, for a very specific use case -- one that could, incidentally, be solved by the user by giving names to these lambdas in the first place. Yes, this punishes the user who has this (IMHO rather funky requirement), but the alternative is to punish everybody by making a generally more complex system.
USA #70
David Haley said:
An internal 'cleanup' might or might not be needed for implementation reasons, but these are precisely implementation details and the user should never, ever have to worry about cleaning up implementation-specific temporaries.

As soon as I 'access' data or 'request' a function's return value, I have the values in hand and they should go away immediately in any other form they might exist in other than the values I have.

You misunderstand me, these ARE implementation details. The user only has to deal with PPI.Load, PPI.Expose, and whatever methods the service exposes. I explained them because it's critical to understand how PPI actually works, in the context of this discussion.

David Haley said:
Quote:
with slightly more complex serialization to accomodate PPI-specific requirements

It's still a little unclear to me, to be honest, if these requirements were driven by actual problems to solve, or problems that could be solved.

I mentioned function callbacks, which have a very clear use that I've mentioned and utilized already. If there's a specific other requirement I mentioned that you don't understand the need for, I'd be glad to address it directly.

David Haley said:
Quote:
but there is no difference between a table containing a table, and a table containing a reference to another table.

Actually, there sort of is. You can think of it as the difference between a containment tree and a containment graph. I think we all agree that we want to be able to pass tables with subtables; it's still unclear to me (and Nick, I believe) why we want tables that can refer to themselves (which is what arbitrary table references allow).

I'm still not understanding, sorry... If you have a table, and you serialize subtables (something we agree we need), we don't want to serialize the same subtable multiple times if it appears more than once (i.e. a graph rather than a tree, as you said). This requires a cache mechanism, which I use. At this point, tables that refer to themselves - or simply cyclic references in general - are valid simply because of the cache. I certainly wasn't thinking of self-referential tables when I wrote the code.

David Haley said:
Quote:
but I found it was easiest to roll my own

A general rule of thumb is to avoid rolling your own unless you really, really have to.


For whatever it's worth, your serialization code (as posted on page 2, v1.0.1, was the most recent I saw in the thread) does not allow a table key to be anything but a string or number, whereas Nick's can (with some limitations). Before you point at the limitations, I would continue to argue that limitations are only a problem insofar as they prevent the tasks we actually have from being accomplished. To be fair, table keys of type 'table' are perhaps weird in the first place, but part of your argument is being language-independent and as general as possible. So this could be a source of confusion. In fact, your serialization function silently ignores stuff it can't serialize, which could lead to very confusing problems!

I hate to say it, but you're several versions behind. I began pasting them elsewhere and linking to them after they got too large for the posts. The latest version is here:

http://mushclient.pastebin.com/f16a36fbc

In relation to things that can't be serialized, my version uses z:~ in its place, which is just a serialized nil.

David Haley said:
TBH, and I hate to harp on this, but it still seems to me like the requirements haven't really been defined. I'm not talking about implementation requirements, I'm talking about API requirements. We're talking of "PPI requirements" but I haven't seen a single place where the high-level goals are cleanly laid out, with real-world use cases and justifications for each of the requirements.

I think in a very, um... different manner than most people I've met. It's very hard for me to lay things out so concretely. I tend to work by examples, such as my ATCP plugin - I posted the new version a day or two ago in its own thread. Most examples are also in my head as I work (and I do throw out ideas that simply aren't useful). If you have any concrete questions about the API, rather than asking me to lay it out for you, I would be happy to oblige. >_>

David Haley said:
Complexity in general, solving problems simply because you can, is not always a good thing. This is a Very Important Concept and perhaps hard to explain succinctly. There's a good quote about perfection by Antoine de Saint-Exupery; paraphrasing, the designer knows that he has achieved perfection not when there is nothing left to add but when there is nothing left to take away. This is extraordinarily relevant to computer science in general and API/protocol/etc. design in particular. Solving all kinds of problems sounds awfully nice, until you try to solve so many that you end up confusing the user when it comes to solving the problems you initially had. This is a somewhat lengthy reference, but I would recommend this document:
http://cacm.acm.org/magazines/2009/5/24646-api-design-matters/fulltext
(Caveat: I've only read about half of it so far, myself)

When I look at PPI, I believe that everything has a very important use case. It all works through the same code system. If I had to, I could limit PPI.Expose to methods again, but I don't see the point: it all works through the same core system. PPI.Expose is like passing a parameter; using a client PPI to access it is like getting the parameter from a function call. It doesn't make sense, to me, to limit it in such a way. It would be adding to the code's burden, rather than removing from it.


David Haley said:
To summarize: I find it very difficult to comment without having a very clear picture of the exact problem we're solving and why it's a problem. This should be general, but more specific than just "sending stuff between plugins" -- that much is obvious enough. ;-)

Like I said earlier, it's kind of hard to speak concretely about it unless I have an applicable question. >_> I know what I expect of PPI, but it's hard for me to communicate it.


David Haley said:
Here's an example: we need to send functions around. Do we really need to send functions that don't have proper names? If we only send properly named functions, then we can dramatically simplify the passing-around of functions; namely, you specify them by name and you're done.

Sorry, what? It doesn't matter if it has a name or not, you're giving the function value itself to PPI. And it does give it a name, or more properly, an identifier. The identifier could have been a string, but numbers lent themselves better to uniqueness: just increment the last one. Strings just didn't seem to fit anyways.

David Haley said:
Yes, yes, it would be nice to construct arbitrary callbacks in Lua using closures etc. and then be able to pass these to VBscript or whatever. But this introduces complexity for the entire library, for a very specific use case -- one that could, incidentally, be solved by the user by giving names to these lambdas in the first place. Yes, this punishes the user who has this (IMHO rather funky requirement), but the alternative is to punish everybody by making a generally more complex system.

Functions are not actually passed between plugins. Identifiers are. Any language could theoretically save this identifier somewhere, and when the user is ready, send that identifier back across with a REQUEST message.
Amended on Tue 19 Jan 2010 05:40 AM by Twisol
USA #71
Quote:
I explained them because it's critical to understand how PPI actually works, in the context of this discussion.

This is the kind of thing that should be made explicit, probably, as you describe things. You have two audiences: (1) people who develop PPI (which is really you, and Nick who will eventually approve something for inclusion into MUSHclient) and (2) people who use it.
You need to motivate all of this for both groups, and different motivations are needed for each.

Quote:
I mentioned function callbacks, which have a very clear use that I've mentioned and utilized already.

These can be achieved without rolling yet another serialization library (see below).

Quote:
If you have a table, and you serialize subtables (something we agree we need), we don't want to serialize the same subtable multiple times if it appears more than once (i.e. a graph rather than a tree, as you said).

If you only have a tree-table structure, you will never have repeated tables. This is what Nick means when he talks about tables containing subtables (tree structure) and self-referential structures (graph structure). (Technically, you can have a graph without self-referential tables; all you need is a repeated table.) What Nick and I are saying is that we don't really see the need for repeated tables.

Quote:
I hate to say it, but you're several versions behind.

The version you linked to does not allow table keys, either. In particular,


    local key = nil
    if type(k) == "string" then
      key = "s:" .. k
    elseif type(k) == "number" then
      key = "n:" .. k
    end

It's pretty apparent that if type(k) is anything other than string or number, 'key' remains nil; looking at the lines right below that, ArraySet is only called when key is not nil. And, it's all silently ignored.

Quote:
If you have any concrete questions about the API, rather than asking me to lay it out for you, I would be happy to oblige.

Sigh. :(
Well, this is complicated then. :) You're trying to convince people that your solution is inherently superior, despite their reservations that it's too complex for the problem at hand. There is really no other way for you to convince this than to "lay it out". :-/

The point is that the questions aren't about the API's details, it's about whether this is the appropriate solution in the first place.

Quote:
Sorry, what? It doesn't matter if it has a name or not, you're giving the function value itself to PPI. And it does give it a name, or more properly, an identifier. The identifier could have been a string, but numbers lent themselves better to uniqueness: just increment the last one. Strings just didn't seem to fit anyways.

This is the kind of stuff I'm talking about... there's really no need to send the function value. It's solving a problem just because you can, not because it's a problem that needs to be solved. It's overly complicated and only helpful in a very, very small number of cases. In the vast majority of cases, the function's name is sufficient, and it considerably simplifies things.
USA #72
David Haley said:

Quote:
Sorry, what? It doesn't matter if it has a name or not, you're giving the function value itself to PPI. And it does give it a name, or more properly, an identifier. The identifier could have been a string, but numbers lent themselves better to uniqueness: just increment the last one. Strings just didn't seem to fit anyways.

This is the kind of stuff I'm talking about... there's really no need to send the function value. It's solving a problem just because you can, not because it's a problem that needs to be solved. It's overly complicated and only helpful in a very, very small number of cases. In the vast majority of cases, the function's name is sufficient, and it considerably simplifies things.

I'll respond to the rest later, because I'm tired atm, but I'm NOT sending its value by any stretch of the imagination. I am storing its value locally within the plugin that's sending it, and sending an identifier that stands in for the function. For all intents and purposes, that is a name. It's not the user's direct name for it, but it is absolutely a name.

EDIT after shower: Serializing a function's actual value would be catastrophic on the function's actual use as a communication device. It would keelhaul upvalues, for one. No, if I serialize its actual value, my above examples wouldn't work, as I explicitly placed an extra variable just for one of the callbacks to print out.

For why I store the function before sending an identifier, it's like this:

a = function() end
b = a
a = nil


'b' is unaffected, no? It still has a valid reference to the function. If I referred directly to the variable the user sent, then by modifying or removing that variable, the plugin it was sent to suddenly has a bad reference. Simply put, I am trying to preserve natural Lua semantics.
Amended on Tue 19 Jan 2010 05:41 PM by Twisol
USA #73
David Haley said:

Quote:
I explained them because it's critical to understand how PPI actually works, in the context of this discussion.

This is the kind of thing that should be made explicit, probably, as you describe things. You have two audiences: (1) people who develop PPI (which is really you, and Nick who will eventually approve something for inclusion into MUSHclient) and (2) people who use it.
You need to motivate all of this for both groups, and different motivations are needed for each.

My apologies, then - I didn't think I was supposed to be catering to users at this point. I was just trying to explain my version to Nick as per his questions.

David Haley said:
Quote:
I mentioned function callbacks, which have a very clear use that I've mentioned and utilized already.

These can be achieved without rolling yet another serialization library (see below).

I need somewhere to put the function and replace it with an identifier, and I have a language-agnostic serialization protocol that needs to be supported. The serialize module wouldn't cut it because of the latter, and I also specifically wanted to limit the types of keys allowed.

David Haley said:
Quote:
If you have a table, and you serialize subtables (something we agree we need), we don't want to serialize the same subtable multiple times if it appears more than once (i.e. a graph rather than a tree, as you said).

If you only have a tree-table structure, you will never have repeated tables. This is what Nick means when he talks about tables containing subtables (tree structure) and self-referential structures (graph structure). (Technically, you can have a graph without self-referential tables; all you need is a repeated table.) What Nick and I are saying is that we don't really see the need for repeated tables.

My goal is to retain normal Lua semantics as much as possible. It was a very simple addition, and I think it is a very useful addition. Furthermore, I needed the cache in order to separate tables out so that they all go in separate variables. You could argue that you could put them all in one variable, but (1) I don't like at all the sound of that approach, it's too hacked; and (2) as more tables are added, more and more backslashes are added by ArrayExport() to escape the | separator in the arrays. Wimpy arguments? Maybe. But there's no gain to doing it that way, either.


David Haley said:
Quote:
I hate to say it, but you're several versions behind.

The version you linked to does not allow table keys, either. In particular,


    local key = nil
    if type(k) == "string" then
      key = "s:" .. k
    elseif type(k) == "number" then
      key = "n:" .. k
    end

It's pretty apparent that if type(k) is anything other than string or number, 'key' remains nil; looking at the lines right below that, ArraySet is only called when key is not nil. And, it's all silently ignored.

Tables as keys is one of those things that I thought to myself and said, no, that's not going to be useful at all in this context. First and foremost, very, very few languages can use an arbitrary type as a key. Secondly, table keys are even more unlikely to be supported. Thirdly, even if I did allow table keys, all passed tables are copies. You would have to run over pairs() in the first place to be able to get at them, which defeats the purpose of tabular keys.

David Haley said:
Quote:
If you have any concrete questions about the API, rather than asking me to lay it out for you, I would be happy to oblige.

Sigh. :(
Well, this is complicated then. :) You're trying to convince people that your solution is inherently superior, despite their reservations that it's too complex for the problem at hand. There is really no other way for you to convince this than to "lay it out". :-/

The point is that the questions aren't about the API's details, it's about whether this is the appropriate solution in the first place.

I would honestly like to ask you what your own solution would be. I get the feeling that maybe I wrote PPI too 'cleverly', which can be a very bad thing, and if you have any ideas I would like to hear them. I'm trying to explain PPI by answering your and Nick's criticisms and questions, because it comes easily to me when responding.
Amended on Tue 19 Jan 2010 06:13 PM by Twisol
USA #74
Quote:
but I'm NOT sending its value by any stretch of the imagination. I am storing its value locally within the plugin that's sending it, and sending an identifier that stands in for the function.

I didn't say you were; I'm saying that this extra indirection is unnecessarily complicated. I do not believe that it is a common use case to want to send a function value rather than the (user-space) name of a function.

For example, you later say: "I need somewhere to put the function and replace it with an identifier".
But this complexity wouldn't exist if we (as users) were only sending function names, as opposed to function values. (That the function values are internally converted to random identifiers is irrelevant at present. As far as the user is concerned, values are being sent.)


Quote:
Simply put, I am trying to preserve natural Lua semantics.

Then this is confusing to me w.r.t. making a language agnostic cross-plugin communication layer.

Is the problem to deal with cross-plugin, cross-language communication, or is the problem to deal with Lua communication?

Quote:
My apologies, then - I didn't think I was supposed to be catering to users at this point.

You are necessarily dealing with (even hypothetical) users as soon as you start talking about designing an API to be included in the MUSHclient "core". An API without users is not a useful project. The argument being made against the existing API is that it seems to complex for what users are likely to need; the response (from you) must argue that users in fact will need this complexity often enough for it to become a requirement.

Quote:
Wimpy arguments? Maybe. But there's no gain to doing it that way, either.

I'm not sure which way you're referring to with "that way", or if you're responding to me in general; I was talking about the unclear value of allowing graph-like tables as opposed to tree-like tables, which simplifies implementation.

Quote:
First and foremost, very, very few languages can use an arbitrary type as a key.

Not really. C++, Java, Perl, Lua, Python, Ruby, to name just a few, all allow tables as keys (with restrictions in some cases).

All I'm trying to say is that it seems somewhat arbitrary to argue for generality in one case and then to get rid of another generality by saying it's specific.

I mean, if we're worried about serializing complex tables with cyclical structures and all that, and especially if we're worrying about maintaining Lua semantics, then in fact it seems very reasonable to want to keep Lua semantics in this case. Otherwise, the API is behaving inconsistently: it's keeping the semantics here, but dropping them there -- and silently dropping them, too!

Quote:
I would honestly like to ask you what your own solution would be.

My honest response is that I don't really know because I do not feel that there has been an adequate presentation of what users of cross-plugin communication actually need. For example, I don't know if a low-level communication layer is what we want, or if we want a higher-level semantics transport layer. My first and strongest gut reaction is to avoid complexity unless it is clear that it is unavoidable. My second gut reaction is to separate the implementation from the concept. There's been a lot of talk about implementation so far, but relatively little talk about the problems we're trying to solve. Examples:
Why do we want to send complex tables?
Why do we want to send function values (remember, from the user's perspective), and not function names?
What are we trying to send in the first place?
Why do we need plugins to talk to each other?

For technical questions, "because we can" is not an acceptable answer unless it adds no complexity to the end result.

I find it remarkably difficult, if not basically impossible, to design a system when you don't know exactly what you're designing to. What are the user's requirements? These are different from an implementation's requirements, which should be driven by the user's requirements in the first place.
Australia Forum Administrator #75
Twisol said:

I was just trying to explain my version to Nick as per his questions.


Actually I said:

Nick Gammon said:

A programming exercise is not really done until it is documented, otherwise people don't know what the code does, how to use it, and why they might want to.


I wasn't referring to myself as "people" (although I am a person, I admit). I meant "people" as in "other people", Peter the Programmer or Dorothy the Developer.

I don't think Peter or Dorothy would care exactly how you serialize, which is effectively what you described in the reply. I meant to describe:

  • What the module does in general
  • Why you might use it instead of CallPlugin
  • What features it provides to the plugin-writer
  • How to use those features
  • Give some realistic examples of use


There has been some talk about function callbacks. I don't believe my simpler version supports that, and until I see an example I'm not even certain what you are referring to.

Do you mean plugin A uses PPI to call function f in plugin B, passing it a function g (the function g being implemented in plugin A) which plugin B then "calls back" once or many times? Or does it mean something else?

If I have got that bit right (and in the absence of examples or documentation I'm not sure I have) then what is the lifetime of this callback? In other words, once f is called can g be called after f returns? Or only during the call to f? Does the callback use PPI for the called-back function call? If so does it recurse to do so? What provision is there for detecting loops (eg. f calls g which calls f, infinitely)?

Can a third plugin get involved? Eg. f calls g which calls h in a third plugin?

Twisol said:

I'm trying to explain PPI by answering your and Nick's criticisms and questions ...


I'm not trying to criticize here, just trying to arrive at the "best" end solution. Of course, the best solution depends a bit on defining the problem as David is trying to say, more than once.

Let me give you an analogy, let's say one of us has a sports car and the other a large truck.

Now the sports car:

  • Will go faster
  • Will be more comfortable inside
  • Can only carry small loads


The truck:

  • Will go slowly
  • Will be less comfortable inside
  • Can carry large loads


Now if I say the my sports car is faster than your truck, it is an observation rather than a criticism. And if the objective is to move a ton of bricks from place A to place B then the truck is the correct solution.

However if the objective is to take your girlfriend out to the movies, then the sports car would be the more correct solution.
Amended on Tue 19 Jan 2010 07:34 PM by Nick Gammon
USA #76
I'm on my iPhone at the moment, so it's hard to write a decent reply. I'll post again later to make up for it.

However, I think my PPI is more of a Hummer (H2). Powerful, carries big loads, and is comfy inside to boot. Impresses the ladies (the ones who don't go crazy about mpg wastefulness, at least).

Quote:
Do you mean plugin A uses PPI to call function f in plugin B, passing it a function g (the function g being implemented in plugin A) which plugin B then "calls back" once or many times? Or does it mean something else?

If I have got that bit right (and in the absence of examples or documentation I'm not sure I have) then what is the lifetime of this callback? In other words, once f is called can g be called after f returns? Or only during the call to f? Does the callback use PPI for the called-back function call? If so does it recurse to do so? What provision is there for detecting loops (eg. f calls g which calls f, infinitely)?


It sounds like you've got it right. Yes, once a function is passed/received, it can be stored and executed at a later date. That's how my ATCP plugin works: clients register a callback against a specific event, and ATCP calls them when it occurs.

Callback do go through the PPI layer, just like any other REQUEST message. There is no facility to detect infinite loops, nor do I think it is feasible to add.
Quote:
Can a third plugin get involved? Eg. f calls g which calls h in a third plugin?

Theoretically, sure. The proxy method that the receiving side gets is still just a regular function.



David, I will respond when I'm home, but it's easier for the user to pass values because the receiving end can work directly with a function object rather than a name. In order to emulate this with Nick's version, you'd need to Expose your method and pass it's name, and both sides would have to agree that it's a function name and not a regular string. It also means any plugin can suddenly easily access the function you intended for just the one target, not to mention clutters the public PPI space.
Amended on Tue 19 Jan 2010 09:00 PM by Twisol
USA #77
Quote:
David, I will respond when I'm home, but it's easier for the user to pass values because the receiving end can work directly with a function object rather than a name.

Except that the other end might have no means of even conceiving of "function objects". A name is a cross-language-portable means of identifying a function, a function object is a much more language-specific concept.

Quote:
In order to emulate this with Nick's version, you'd need to Expose your method and pass it's name, and both sides would have to agree that it's a function name and not a regular string.

I don't see why it's a problem to assume that a string used as the function name is referring to a function and not a regular string.

In other words, if, as a user, I say:
"Call function F with parameters X, Y, Z..."
it seems that there is no "agreeing" to be done beyond the obvious that "F" is a function name.

As for callbacks, well, presumably you will do something like:

"Call function 'CoolPlugin.RegisterCallback' with parameter 'MyNiftyCallback'"

at which point there is also no "agreeing" to be done beyond the straightforward convention.

In other words: why do we need the most general possible method of passing around function values, when the use cases (given so far) for passing functions are actually quite simple?

Quote:
It also means any plugin can suddenly easily access the function you intended for just the one target, not to mention clutters the public PPI space.

It's more or less a "shared function" anyhow, so I'm not sure why this is a problem. If it's really a problem, security can be introduced by obscurity and the name can be obfuscated in some funky way.

If you need to give a function to just one plugin and no other, this can also be achieved by adding a parameter specifying the calling plugin, and you can then restrict access to symbols by looking at that parameter. I'm not convinced this is actually useful, but it's still simpler than all kinds of state-tracking.

I'm also not sure if this is a general statement of the general problem, or a statement made w.r.t. your current implementation. Are you saying that this necessarily clutters any possible implementation of cross-plugin communication, or that this is the case of your current implementation?
USA #78
Not home yet, battery low.


David, in no way am I sending a function object. It can look that way to the user, in this Lua version, because it's intuitive. But I am only ever sending a name, an identifier. Another language need not have the same user interface my Lua PPI does; it need not turn this identifier back into a functional object. Lua PPI does this because it is intuitive and it makes sense.

I would continually like to point to my ATCP plugin, and a client Roomname plugin, both of which have been posted recently in another thread.
USA #79
David Haley said:
Quote:
First and foremost, very, very few languages can use an arbitrary type as a key.

Not really. C++, Java, Perl, Lua, Python, Ruby, to name just a few, all allow tables as keys (with restrictions in some cases).

All I'm trying to say is that it seems somewhat arbitrary to argue for generality in one case and then to get rid of another generality by saying it's specific.

I mean, if we're worried about serializing complex tables with cyclical structures and all that, and especially if we're worrying about maintaining Lua semantics, then in fact it seems very reasonable to want to keep Lua semantics in this case. Otherwise, the API is behaving inconsistently: it's keeping the semantics here, but dropping them there -- and silently dropping them, too!

C++ and Java are not MUSHclient scripting languages. Lua is the one we're already discussing. That said, I didn't know that Perl, Python and Ruby allowed table keys. However, I still stand by my reasoning that there is no valid use case for tables as keys. I could include them if I wanted, simply by adding another conditional to the serialization routines - if you think I should, it's an easy enough addition - but I don't see the point.


David Haley said:
Quote:
I would honestly like to ask you what your own solution would be.

My honest response is that I don't really know because I do not feel that there has been an adequate presentation of what users of cross-plugin communication actually need. For example, I don't know if a low-level communication layer is what we want, or if we want a higher-level semantics transport layer. My first and strongest gut reaction is to avoid complexity unless it is clear that it is unavoidable. My second gut reaction is to separate the implementation from the concept.

We differ at a fundamental level here: I don't believe this is complicated. It rests on three core messages (each with a very clear purpose), a serialization protocol (aimed to be language-agnostic and extensible), and certain allowable data types. Everything else just drives it.

David Haley said:
There's been a lot of talk about implementation so far, but relatively little talk about the problems we're trying to solve. Examples:
1. Why do we want to send complex tables?
2. Why do we want to send function values (remember, from the user's perspective), and not function names?
3. What are we trying to send in the first place?
4. Why do we need plugins to talk to each other?

1. I assume you mean 'complex' in a more scientific meaning. I think it is perfectly reasonable to think that you might send a list where the same table appears twice. Are you saying there are no occasions when this might be warranted? Besides, I needed some form of tracking to make sure every table got its own variable. I gave my reasons for this separation-by-table previously.

2. It sounds like this question contradicts itself. We're not sending function values, as you noted, just an identifier. It makes sense, for this Lua version of PPI, to provide a natural, feels-like-Lua interface. Pass a function, get a function, it all feels like Lua. That's the point. A non-Lua PPI can wrap around the serialization protocol however it wants, and provide its own user interface. It's separation of concerns: the protocol is as language agnostic as I personally can understand it, and the implementation is as intuitive as possible, which means close coupling with the language.

3. I can't answer that properly without an actual use scenario. At a fundamental level, we're just sending lists of data. In the case of ATCP, the client registers intent with the service, and the service pushes data back to the client later. I'm sorry if I misunderstand the question. I just want PPI to be a general-use transportation library to allow scripts to interact more fluidly than raw CallPlugin.

4. Because there are some resources which simply cannot or should not be accessed by multiple threads. ATCP data is one. If multiple plugins try to pull ATCP data from the packets, all but one will starve. As another example, you have the typical MUD prompt. If multiple prompt triggers gag it and try to re-echo it out differently, you suddenly have multiple prompts. Having a prompt plugin would allow you to centralize your prompt modifications, using PPI to communicate with the plugin. There are many other examples of resources like this.

Of course, special resources aren't the only use case. Another guy I know wants to create a combat system broken up into plugins, using PPI to communicate with a central core (which might have an event loop, for example). The client plugins could be 'class' plugins, which implement triggers specific to certain class attacks. The core would be the driving force. He's using CallPlugin with all sorts of junk right now, which is driving me crazy. It's ugly and IMO hard to maintain (having seen the code). PPI - specifically my version, with functions as valid parameters - would immensely clear up his code to no end.

David Haley said:
For technical questions, "because we can" is not an acceptable answer unless it adds no complexity to the end result.

I tried my best to avoid 'because we can' in my above answers. Thank you for the questions. :)

David Haley said:
I find it remarkably difficult, if not basically impossible, to design a system when you don't know exactly what you're designing to. What are the user's requirements? These are different from an implementation's requirements, which should be driven by the user's requirements in the first place.

I'd like to think my answer to question 4 helped clear this up. If not, please say so!



Nick, next reply will be to you.



Also, Avatar is amazing. Go see it.
USA #80
Nick Gammon said:
Twisol said:

I'm trying to explain PPI by answering your and Nick's criticisms and questions ...

I'm not trying to criticize here, just trying to arrive at the "best" end solution. Of course, the best solution depends a bit on defining the problem as David is trying to say, more than once.

No, no, I understand. It's all constructive. Do excuse me if I sound frustrated sometimes... I am, but I don't mind. ;)

Nick Gammon said:
*What the module does in general
*Why you might use it instead of CallPlugin
*What features it provides to the plugin-writer
*How to use those features
*Give some realistic examples of use


PPI is a module intended to simplify design and implementation of complex systems. On the surface, it facilitates clear syntax and semantics native to Lua (the port we're discussing), allowing you to 'expose' arbitrary values to other plugins. A 'service' Expose()s these values, while a 'client' simply Load()s a PPI interface table and accesses these values as though they were actually in the table. Supported data types are: string, number, boolean, table, function, and nil. All data retrieved is copied, not referenced; modifications to the data will not be reflected at its source.

Function values may be executed just like any other function. Any of the aforementioned values can be passed, and any number of parameters may be sent. Again, all data passed (and returned) are copies of the original.

As PPI implements such a full-featured interface, its semantics are nearly identical with Lua's own, providing a much more intuitive method of communicating between plugins. Where CallPlugin() only accepts one (string) parameter, to a given global method, PPI accepts any number of parameters, to a method that need not be global, only exposed (whether that be by Expose(), or by returning one from a PPI-invoked method). It also supports directly accessing values, rather than calling a method to retrieve it, which feels more native - the difference between CallPlugin("fe19016faddd8daa3f862627", "GetVersion", "") and foobar.version.

To expose a value, use PPI.Expose("identifier", value). The identifier will be used by other plugins to access the value.

To access a value, a client must first use foobar = PPI.Load("plugin's ID"), then access the returned table using foobar.identifier.

PPI is largely intended for use in complex multi-plugin systems, ranging from protected resources (ATCP, ZMP, or the prompt) to modular combat systems, and anywhere in-between. It also is very easy to understand as it uses native Lua syntax and semantics.
Amended on Wed 20 Jan 2010 02:41 AM by Twisol
USA #81
Quote:
David, in no way am I sending a function object. It can look that way to the user, in this Lua version, because it's intuitive.

Again, I never said you were actually sending function objects. It doesn't matter what you're sending under the hood as long as the interface is that you pass a function object. For all intents and purposes, as far as the user is concerned, you are sending function objects -- and your API must deal with receiving function objects as parameters to be sent around.

Quote:
Lua is the one we're already discussing.

No it's not -- it's the language you happen to have implemented this in so far. But a constantly recurring them is language independence, and the ability to reimplement in any scripting language. Therefore Lua's particularities aren't really relevant.

Quote:
However, I still stand by my reasoning that there is no valid use case for tables as keys.

You've basically just asserted this, but when other people have stated that they don't really see valid use cases for other things, you've basically just asserted that they're wrong. This makes it hard to make forward progress. Clearly if people aren't convinced, then you need more to be more convincing or reevaluate your position. I'm not really sure how else to say it. (And this isn't meant to mean offense!)

Quote:
I could include them if I wanted, simply by adding another conditional to the serialization routines - if you think I should, it's an easy enough addition - but I don't see the point.

I don't see the point either, it's added complexity for extraordinarily infrequent utility. The point was only that there seems to have been a somewhat arbitrary line drawn when it comes to supporting this or that generality.

Quote:
We differ at a fundamental level here: I don't believe this is complicated. It rests on three core messages (each with a very clear purpose), a serialization protocol (aimed to be language-agnostic and extensible), and certain allowable data types. Everything else just drives it.

Complicated doesn't mean hard to understand (even though you have said yourself that some things are confusing, like that double-storage thing Nick discussed). It means that stuff is happening that doesn't necessarily need to happen.

And we've already seen that we're actually not all that language-agnostic anyhow.

Quote:
I think it is perfectly reasonable to think that you might send a list where the same table appears twice. Are you saying there are no occasions when this might be warranted?

I will not say that one would never, ever want to do this. But then again, I could say that about a very great deal of things: including tables as keys. My argument is not that you would never want to send complex tables; my argument is that you would want to so rarely that it does not justify increasing the general complexity to deal with this.

Quote:
We're not sending function values, as you noted, just an identifier. It makes sense, for this Lua version of PPI, to provide a natural, feels-like-Lua interface.

We're talking about a language-agnostic protocol, in which one plugin can call some arbitrary code in another plugin. The convention that all functions must be passed by their actual name is simple to understand and the most likely to be as portable as possible. Furthermore, if I'm calling out to another plugin, I don't call FunctionWithRandomIdentifierXYZ, I call a specific name. Passing callbacks (or any other function) by name (again, not random identifier) is consistent with calling functions by name.

Now, as a convenience, you could accept functions by value and use debug.getinfo to introspect a useful name, and complain if one cannot be found.

But this is a good example of the distinction to be found between the core of a design and conveniences, or syntactic sugar if you will, built on top of it.
This convenience requires an extremely simple type check followed by a debug.getinfo call, and entirely obviates all the complexity of generating and storing random function identifiers.

Quote:
I can't answer that properly without an actual use scenario.

This isn't meant to be sarcastic or facetious, but how can we possibly design a reasonable, practical API without having actual use scenarios? It's very hard to design things in a void, from theory alone.

It's possible that you're trying to be too general here, which is I think the point that Nick was making. Complexity is warranted if it's addressing real problems. Complexity is not warranted if it's solving problems simply because they can be solved (even if those problems are relatively easy to solve, because it's extraordinarily rare that you can make one change without impacting the rest of the system).

It's a funny fact of CS in general that often, solving more can actually provide less value. There's a very fine line between solving the right amount, and solving too much. After years of doing this, I still find this line hard to navigate. One thing I do know is that as people accumulate years, they tend to get more and more wary of complexity unless it's necessary.

Quote:
4. Because there are some resources which simply cannot or should not be accessed by multiple threads. ATCP data is one. If multiple plugins try to pull ATCP data from the packets, all but one will starve. As another example, you have the typical MUD prompt. If multiple prompt triggers gag it and try to re-echo it out differently, you suddenly have multiple prompts. Having a prompt plugin would allow you to centralize your prompt modifications, using PPI to communicate with the plugin. There are many other examples of resources like this.

These are generally good examples (although the prompt one sounds like you're doing all the stuff in one plugin, so I'm not sure where the cross-plugin communication comes in). To be clear, I also completely believe that being able to communicate between plugins is a Very Useful Thing. The point of the exercise was to discover the essence of what cross-plugin communication is really about. The ATCP example, for instance, only needs to shuffle around very simple data. Similarly for the prompt plugin. So far, we haven't really justified anything more complex than numbers, strings, and maybe basic key:value maps from numbers/strings to numbers/strings. We have perhaps also justified simple callbacks, so that the centralized handler of a shared resource can notify its (potentially multiple) listeners that new data is ready.

Quote:
Of course, special resources aren't the only use case. Another guy I know wants to create a combat system broken up into plugins, using PPI to communicate with a central core (which might have an event loop, for example). The client plugins could be 'class' plugins, which implement triggers specific to certain class attacks. The core would be the driving force.

Why does stuff need to be separated into several plugins, as opposed to one plugin with several modules? If the other plugins cannot exist without the core, and the core is useless without the others, it's hard to justify the existence of several plugins. (The existence of several modules is however easily justifiable.)

Quote:
PPI - specifically my version, with functions as valid parameters - would immensely clear up his code to no end.

It could also be that his code is just poorly designed, and could be refactored without bringing this considerable tool to bear.

We spoke earlier about encouraging behavior that we might not like. This could be an example. By allowing plugins to send completely arbitrary data, we make it easy (or at least easier) to break stuff into separate plugins when modules might be more appropriate. If the cross-plugin transport layer has clearly defined goals (of which arbitrary communication is not one) people need to think more. In other words, a clear and concise design can propagate upwards to some extent. It's not a good idea, IMO, to force a more-complex API simply because some users are lazy and don't feel like refactoring stuff.

You might argue that there are cases where it reallyreally makes sense to split this stuff into separate plugins and not modules, and where you reallyreally need all these advanced features (along with advanced implementation) -- this is what I'd like to see.


Perhaps some (non-exhaustive) specific questions:
- Why decide that some generality is desired, like complex tables, but other isn't, like more types as table keys? (Why not bools? What about (light)userdata? Where does it stop?)
- Why do we need to support arbitrary function objects at the implementation (not interface) level?
- Are we truly language-agnostic, or have we discovered that maybe we like some things about Lua in particular? If we want to be language-agnostic, how much Lua-specific stuff have we added? How much complexity has been added to other languages' implementations to deal with this stuff? What about things that other languages might like to do, such that new support must be added to Lua? (If your expertise is in Lua, and not Perl/Ruby/Python, how do you really know that you are language-agnostic? How do we know that, with this large system, we won't be adding even more maintenance work as we start expanding to other languages?)

(That second point speaks to the problem of interface vs. implementation: if your interface wants to be convenient that's fine, but that shouldn't necessarily affect the general implementation of the core protocol.)

Regarding the last point, this statement greatly worries me:
Quote:
As PPI implements such a full-featured interface, its semantics are nearly identical with Lua's own, providing a much more intuitive method of communicating between plugins.

This sounds like it will make implementing PPI semantics in other languages very difficult if Lua's semantics don't map well to those other languages. A simple example is that you have no guarantee whatsoever that other languages can even conceive of passing around function values to be called. The PPI API and its very capabilities would then be remarkably different if you are working in Lua vs. VBScript, for instance.
Amended on Wed 20 Jan 2010 04:08 AM by David Haley
USA #82
David Haley said:
Quote:
David, in no way am I sending a function object. It can look that way to the user, in this Lua version, because it's intuitive.

Again, I never said you were actually sending function objects. It doesn't matter what you're sending under the hood as long as the interface is that you pass a function object. For all intents and purposes, as far as the user is concerned, you are sending function objects -- and your API must deal with receiving function objects as parameters to be sent around.

The API does handle this, it just stores them in the same PPI table as any other value. The only special treatment they get is in relation to serialization, where they get their own identifier and are added to the table of callable methods.

David Haley said:
Quote:
Lua is the one we're already discussing.

No it's not -- it's the language you happen to have implemented this in so far. But a constantly recurring them is language independence, and the ability to reimplement in any scripting language. Therefore Lua's particularities aren't really relevant.

PPI, to me, has two distinct parts, which I touched on in my last post: the protocol, and the implementation. The protocol is the backbone of PPI, no matter what language you're working in. The implementation strives to expose the protocol data in as native a way as possible. It doesn't matter how it does it, it's specific to whatever language we're dealing with.

David Haley said:
Quote:
However, I still stand by my reasoning that there is no valid use case for tables as keys.

You've basically just asserted this, but when other people have stated that they don't really see valid use cases for other things, you've basically just asserted that they're wrong. This makes it hard to make forward progress. Clearly if people aren't convinced, then you need more to be more convincing or reevaluate your position. I'm not really sure how else to say it. (And this isn't meant to mean offense!)

I've tried to give use cases as best as I could. When it comes to serializing 'graphs', I'm not really understanding the opposition. It adds no complexity on its own, because most bits it would need have other reasons for being implemented. It just seems completely incorrect, wrong, and unintuitive to serialize the same table twice, resulting in an incorrect model after deserialization. It would would actually add to the user's burden as well as the code if I were to add this restriction.

David Haley said:
Quote:
I could include them if I wanted, simply by adding another conditional to the serialization routines - if you think I should, it's an easy enough addition - but I don't see the point.

I don't see the point either, it's added complexity for extraordinarily infrequent utility. The point was only that there seems to have been a somewhat arbitrary line drawn when it comes to supporting this or that generality.

The difference between table keys (which I don't want) and table 'graphs' (which I do want) is that in both cases, changing it in the opposite direction would add more code and an extra burden on the user.

David Haley said:
Quote:
We differ at a fundamental level here: I don't believe this is complicated. It rests on three core messages (each with a very clear purpose), a serialization protocol (aimed to be language-agnostic and extensible), and certain allowable data types. Everything else just drives it.

Complicated doesn't mean hard to understand (even though you have said yourself that some things are confusing, like that double-storage thing Nick discussed). It means that stuff is happening that doesn't necessarily need to happen.

And we've already seen that we're actually not all that language-agnostic anyhow.

The double-storage thing is an implementation detail specific to Lua so that I can get at the private data from within __index, based on the PPI table being accessed. It's not that confusing once you understand that.

I'm not sure what stuff is happening that doesn't really need to happen, though. Like I've said, everything is based around a core system of three messages, serialization, and saving exposed values so clients can reach them. Function paramters/returns use the same system as PPI-exposed functions, even.

David Haley said:
Quote:
I think it is perfectly reasonable to think that you might send a list where the same table appears twice. Are you saying there are no occasions when this might be warranted?

I will not say that one would never, ever want to do this. But then again, I could say that about a very great deal of things: including tables as keys. My argument is not that you would never want to send complex tables; my argument is that you would want to so rarely that it does not justify increasing the general complexity to deal with this.

It doesn't increase the complexity, though. I tried to write the serialization routines as simply as possible. Everything in them has its place. I could remove the caching functionality for tables, but that would add to the complexity of the serialization routines.

David Haley said:
Quote:
We're not sending function values, as you noted, just an identifier. It makes sense, for this Lua version of PPI, to provide a natural, feels-like-Lua interface.

We're talking about a language-agnostic protocol, in which one plugin can call some arbitrary code in another plugin. The convention that all functions must be passed by their actual name is simple to understand and the most likely to be as portable as possible. Furthermore, if I'm calling out to another plugin, I don't call FunctionWithRandomIdentifierXYZ, I call a specific name. Passing callbacks (or any other function) by name (again, not random identifier) is consistent with calling functions by name.

You can only ever get at a function through the identifier passed from the ACCESS message (i.e. tbl.someMethod). If you really, really wanted me to, I could change it so that it would use its real name out of the PPI table. But that would, again, create more complexity in the code just to make it look nice. The protocol's not supposed to be readable, it's supposed to be easy to parse and deserialize. Given an identifier - any identifier, be it name or number - you simply store the identifier somewhere and return a proxy of some kind. (This is assuming Lua-style semantics; a PPI in VBscript would undoubtedly have an entirely different user interface) The proxy later takes that same identifier and makes a REQUEST with it. That's all there is to it.

ACCESS and REQUEST (the latter of which should really be called INVOKE) have direct analogues in the __index and __call metamethods. One gets a value from a table, and the other executes a function. __index does not return a name, nor does it execute the function. It just returns something you can use to invoke that function, whether now or later. It's the same idea.

David Haley said:
Now, as a convenience, you could accept functions by value and use debug.getinfo to introspect a useful name, and complain if one cannot be found.

But this is a good example of the distinction to be found between the core of a design and conveniences, or syntactic sugar if you will, built on top of it.
This convenience requires an extremely simple type check followed by a debug.getinfo call, and entirely obviates all the complexity of generating and storing random function identifiers.

Why? This is needless complexity on its own; I'm doing pretty much exactly the same thing as __index and __call (see above).

David Haley said:
Quote:
I can't answer that properly without an actual use scenario.

This isn't meant to be sarcastic or facetious, but how can we possibly design a reasonable, practical API without having actual use scenarios? It's very hard to design things in a void, from theory alone.

No, heh, I meant that your question is hard to answer without the context of a specific scenario. It's your question that's in the void.

David Haley said:
It's possible that you're trying to be too general here, which is I think the point that Nick was making. Complexity is warranted if it's addressing real problems. Complexity is not warranted if it's solving problems simply because they can be solved (even if those problems are relatively easy to solve, because it's extraordinarily rare that you can make one change without impacting the rest of the system).

It's a funny fact of CS in general that often, solving more can actually provide less value. There's a very fine line between solving the right amount, and solving too much. After years of doing this, I still find this line hard to navigate. One thing I do know is that as people accumulate years, they tend to get more and more wary of complexity unless it's necessary.

See my previous answers. Everything I've written in PPI has a distinct purpose and reasoning. It's probably not obvious to everyone reading it, that's why we're all here right now.

I'd like to mention, by the way, that normally if someone says something is wrong with my code or design, I'm very reasonable about it. More often than not, I make lots of changes based on their feedback. I'm certainly not as defensive as I appear here! The problem, as I feel it, is that you're not understanding my reasoning in PPI (and I've put a lot more thought and reasoning into it than many of my previous projects, for what that's worth). I think you may have a point with the graphs, but I've also given my reasoning on why I think it's best that way. It's just frustrating because everything makes so much sense to me, and when I read your and Nick's posts, I (rhetorically) wonder if you've (a general 'you') read over the code, tried what I've explained, or seen my working examples (ATCP and roomname). I understand that Nick has, and that you've looked over PPI at least, but again... we're back to not understanding the logic in the code. (That's probably partially my fault.)

... Eh, no offense meant! I just really want to make sure you understand my point of view. Not trying to start a flame war or anything. =/

(EDIT: What I'm trying to say, I think, is: please don't think I'm throwing a tantrum or something because I'm not getting my way. That's absolutely not it. :( )

David Haley said:
Quote:
4. Because there are some resources which simply cannot or should not be accessed by multiple threads. ATCP data is one. If multiple plugins try to pull ATCP data from the packets, all but one will starve. As another example, you have the typical MUD prompt. If multiple prompt triggers gag it and try to re-echo it out differently, you suddenly have multiple prompts. Having a prompt plugin would allow you to centralize your prompt modifications, using PPI to communicate with the plugin. There are many other examples of resources like this.

These are generally good examples (although the prompt one sounds like you're doing all the stuff in one plugin, so I'm not sure where the cross-plugin communication comes in). To be clear, I also completely believe that being able to communicate between plugins is a Very Useful Thing. The point of the exercise was to discover the essence of what cross-plugin communication is really about. The ATCP example, for instance, only needs to shuffle around very simple data. Similarly for the prompt plugin. So far, we haven't really justified anything more complex than numbers, strings, and maybe basic key:value maps from numbers/strings to numbers/strings. We have perhaps also justified simple callbacks, so that the centralized handler of a shared resource can notify its (potentially multiple) listeners that new data is ready.

In relation to the prompt one, my concept is more about allowing other plugins to contribute bits of data to the prompt plugin, and allowing the user to pick and choose what data to display. Something along those lines, at least - it's one of those ideas that I never really got a chance to work with.

David Haley said:
Quote:
Of course, special resources aren't the only use case. Another guy I know wants to create a combat system broken up into plugins, using PPI to communicate with a central core (which might have an event loop, for example). The client plugins could be 'class' plugins, which implement triggers specific to certain class attacks. The core would be the driving force.

Why does stuff need to be separated into several plugins, as opposed to one plugin with several modules? If the other plugins cannot exist without the core, and the core is useless without the others, it's hard to justify the existence of several plugins. (The existence of several modules is however easily justifiable.)

It's easier to unload or mix/match plugins than it is modules, IMO. The core, by the way, is by no means useless without the extra plugins. That's sort of the point. You have outer modules which represent functionality specific to responding to a certain class's afflictions/abilities. You disable/enable them depending on what classes you're fighting (yes, multiple, especially in raid situations), which can be important for anti-illusion mechanisms (illusions allow you to room-emote text, which can mess up your system if they illusion various attack/affliction messages).

It also helps if you want to support addons, i.e. plugins that can access and interact with the core. The popular Vadi System (a proxy system rather than a client-specific one) allows you to create separate DLLs and load them as addons.

David Haley said:
Quote:
PPI - specifically my version, with functions as valid parameters - would immensely clear up his code to no end.

It could also be that his code is just poorly designed, and could be refactored without bringing this considerable tool to bear.

We spoke earlier about encouraging behavior that we might not like. This could be an example. By allowing plugins to send completely arbitrary data, we make it easy (or at least easier) to break stuff into separate plugins when modules might be more appropriate. If the cross-plugin transport layer has clearly defined goals (of which arbitrary communication is not one) people need to think more. In other words, a clear and concise design can propagate upwards to some extent. It's not a good idea, IMO, to force a more-complex API simply because some users are lazy and don't feel like refactoring stuff.

Quite possible, even perhaps probable. But see above for why I don't think it's a bad design.

David Haley said:
Perhaps some (non-exhaustive) specific questions:
- Why decide that some generality is desired, like complex tables, but other isn't, like more types as table keys? (Why not bools? What about (light)userdata? Where does it stop?)

Well, I could add bools. The problem with table keys is that those keys are the only copy in that whole environment, because all parameters are copied. As I mentioned previously, you'd have to do a pairs() loop just to get at the keys to use them, which is rather pointless - and sending a table key -back- to the source plugin would be useless, because it's yet another copy, so the source plugin wouldn't be able to access the values it's supposed to. Booleans don't suffer this issue.

David Haley said:
- Why do we need to support arbitrary function objects at the implementation (not interface) level?

I'm not sure what you're saying. I always saw PPI as having two levels: protocol and implementation. The implementation is necessarily locked to a specific language. Other languages could take a completely different approach to implementing a PPI interface in order to give that particular language a more intuitive feel. As I have it, the implementation is tightly coupled with the interface for that very reason.

The concept of a function identifier, at the protocol level, is fairly language-agnostic. Any language should be able to represent it in some manner, even if it's nothing like the Lua approach.

David Haley said:
- Are we truly language-agnostic, or have we discovered that maybe we like some things about Lua in particular? If we want to be language-agnostic, how much Lua-specific stuff have we added? How much complexity has been added to other languages' implementations to deal with this stuff? What about things that other languages might like to do, such that new support must be added to Lua? (If your expertise is in Lua, and not Perl/Ruby/Python, how do you really know that you are language-agnostic? How do we know that, with this large system, we won't be adding even more maintenance work as we start expanding to other languages?)

Again, the implementation itself is necessarily locked to a language. The only part that you absolutely have to have is a compatible set of serialization routines, which act as the decoupling filter between implementation and protocol.

David Haley said:
(That second point speaks to the problem of interface vs. implementation: if your interface wants to be convenient that's fine, but that shouldn't necessarily affect the general implementation of the core protocol.)

See above. (I'm only being brief because I don't want you to think I'm ignoring parts of your post by not including them.)

David Haley said:
Regarding the last point, this statement greatly worries me:
Quote:
As PPI implements such a full-featured interface, its semantics are nearly identical with Lua's own, providing a much more intuitive method of communicating between plugins.

This sounds like it will make implementing PPI semantics in other languages very difficult if Lua's semantics don't map well to those other languages. A simple example is that you have no guarantee whatsoever that other languages can even conceive of passing around function values to be called. The PPI API and its very capabilities would then be remarkably different if you are working in Lua vs. VBScript, for instance.

Wrong, as I see it. The serialization routines themselves can be written in pretty much any supported MUSHclient language, IIRC. The actual representation of the protocol data will necessarily be different depending on the language in use, and as such, the implementation/interface would necessarily be different. The API might be remarkably different given a different language, and likely the implementation too (in a language such as VBScript, for example), but the only important part is that they can understand the protocol.

I would write a Ruby port if I understood Ruby well enough, but I haven't used Ruby enough and probably would do a shoddy job of it.
Amended on Wed 20 Jan 2010 06:44 AM by Twisol
Netherlands #83
Good lord, you all are spammy. :)

Since I really am not willing to study the plugin to the amount of depth it is being discussed in, I will bring up another matter instead:

All pre-supplied plugins tend to be useless for experienced plugin writers and new users alike.

Why? Everything is in the same directory, there is no real glossary what each plugin does other than opening them one by one, and at least half of them are highly specific tasks showing how to do one thing that most people aren't interested in, or do not know they are interested in.

The few plugins that might be useful, unmodified and all, are lost in the trees that make up the forest. Mostly because while they are awesome, they lack examples and documentation to point the users in the right direction.

For examples, I do NOT see the practical use of your PPI script at all. Does it solve a problem? Yes. It is even a problem I already solved well enough for my own purposes in the past, which is a far far simpler method than yours and admittedly far less functional.

In a nutshell, what does it do? It sets up a more advanced interface for cross-plugin communication, expanding CallPlugin to the point where you can pass more complicated types on both sides.

I ask you: when is it useful? The way I see it, it has far more limitations than you would like to think. Most of the people I know that dabble with MUSH scripting have trouble understanding how to adjust a trigger in a plugin, nevermind comprehend inter-plugin communication or why they would need your script more-so than plain CallPlugin (if they can even find the latter).

I will make a bold statement and say that this script should not be included in MUSHclient. The only people that will use it are people that need it for a particular plugin that does make use of it, in which case it will always be distributed along because you have a newer version or a specific need for an older version, etc.

Everyone else will either roll their own without even considering to look for something to abstract things away, or they will consciously come looking for it on the forums. (The latter due to the high noise:useful ratio in the supplied examples with MUSHclient.)

So please, maintain a post on the forums, maybe have it stickied or something along those lines so the people who script plugins can find it, but don't include it in MUSHclient.
USA #84
Quote:
PPI, to me, has two distinct parts, which I touched on in my last post: the protocol, and the implementation.

Then you need to clearly define the difference between these and lay out exactly what the protocol is, so that we're all speaking the same language. You keep saying that things are language-agnostic and yet all these Lua-specific points keep creeping in. It makes it very hard to understand where the language-agnostic protocol is. How exactly is a language-independent function callback to be passed around?

Quote:
It just seems completely incorrect, wrong, and unintuitive to serialize the same table twice, resulting in an incorrect model after deserialization. It would would actually add to the user's burden as well as the code if I were to add this restriction.

It only adds to the user's burden if they actually care about the feature. You've made a very strong assumption that these complex tables are utterly necessary to plugin development. Can you give an example of real-world data that requires complex tables?

Quote:
The difference between table keys (which I don't want) and table 'graphs' (which I do want) is that in both cases, changing it in the opposite direction would add more code and an extra burden on the user.

What is the criterion for "wanting" something?

Quote:
The double-storage thing is an implementation detail specific to Lua so that I can get at the private data from within __index, based on the PPI table being accessed. It's not that confusing once you understand that.

Nothing, really, is confusing once you understand it. So this isn't really a justification for complexity.

Quote:
I could remove the caching functionality for tables, but that would add to the complexity of the serialization routines.

How can removing code and features increase complexity?

Quote:
If you really, really wanted me to, I could change it so that it would use its real name out of the PPI table. But that would, again, create more complexity in the code just to make it look nice.

It would only adds complexity because you're trying to shoehorn it into an already complex solution; my suggestion was that the simpler solution should be the solution.

By the way, the point here is not in any fashion to make things "look nice". That is irrelevant.

Quote:
ACCESS and REQUEST (the latter of which should really be called INVOKE)

Then call it invoke; this is still the design phase of the protocol, and we already have irreparable backwards-compatibility brokenness to work around? ;-)

Quote:
Everything I've written in PPI has a distinct purpose and reasoning. It's probably not obvious to everyone reading it, that's why we're all here right now. <etc.>

I don't disbelieve that you have thought this through. But I do think that you have (with the proof-in-the-pudding, as it were) not motivated your decisions clearly enough.

By the way, you mentioned looking through the code. The issue here isn't only complexity of code, it's complexity of the idea or interface.

Quote:
It's easier to unload or mix/match plugins than it is modules, IMO.

I don't know anything about this particular plugin nor frankly do I really care to delve into it. My point was just that this might be using plugins in a way they might not have originally been intended to be used. Or maybe it's a fine usage. Even so, assuming this is perfectly fine, you need to motivate the requirement for complex communication in this instance, as more than just simple data types.

Quote:
The problem with table keys is that those keys are the only copy in that whole environment, because all parameters are copied. As I mentioned previously, you'd have to do a pairs() loop just to get at the keys to use them, which is rather pointless - and sending a table key -back- to the source plugin would be useless, because it's yet another copy, so the source plugin wouldn't be able to access the values it's supposed to.

This is a Lua-specific restriction. Python dictionaries-as-keys don't suffer from this problem. :-)
So why does Lua drive the language-agnostic protocol but not other languages?

Quote:
The concept of a function identifier, at the protocol level, is fairly language-agnostic. Any language should be able to represent it in some manner, even if it's nothing like the Lua approach.

This is true, except that the extra complexity of generating random function identifiers is only useful to languages that have functions as anything more than names in the global namespace. And now there is inconsistency, because Lua plugins will send out identifiers as random stuff, whereas plugins in, say, VBScript will send out identifiers as actual names.

Quote:
(I'm only being brief because I don't want you to think I'm ignoring parts of your post by not including them.)

That's fine; it's easier to just skip something to reduce clutter. If I think you ignored a point and didn't respond to it at all, I'll bring it up again. :P
USA #85
This isn't a plugin, Worstje, it's a library. I tend to completely agree with you when it comes to the pre-supplied plugins, to be honest. But libraries tend to be different. Everyone knows about 'tprint', for example. The 'serialize' module is also probably one of the more used. And 'addxml' definitely makes it a lot easier to dynamically add triggers, aliases, etc. All of them are documented and mentioned on these forums, too. When it comes to libraries - especially the most useful ones, which I would like to think PPI is - it's easier to include them so people don't have to make an extra download, whether you're a developer or a user.

Plugin developers tend to have to cater to those who "have trouble understanding how to adjust a trigger in a plugin" anyways, and I speak from experience! But one really, really good thing about PPI is that it allows, no, encourages cooperation and collaboration between plugin authors. For example, my ATCP plugin. It now uses PPI to make it much easier to communicate with the plugin, allowing the ATCP plugin to push data to only those who want that specific kind of data. It also exposes a Send method which you can use to send an ATCP message to the server. Do other developers need to know how ATCP works? No. But the simplicity of my Roomname plugin is testament to the ease of use PPI supports, without needing to know exactly how PPI or ATCP works.
USA #86
David Haley said:

Quote:
PPI, to me, has two distinct parts, which I touched on in my last post: the protocol, and the implementation.

Then you need to clearly define the difference between these and lay out exactly what the protocol is, so that we're all speaking the same language. You keep saying that things are language-agnostic and yet all these Lua-specific points keep creeping in. It makes it very hard to understand where the language-agnostic protocol is. How exactly is a language-independent function callback to be passed around?


f:1. This is a serialized function, if you want to call it that. 'f' just denotes that this is a function-type parameter. '1' is the PPI identifier of the function. Any language can save that identifier somewhere, wrap a shim around it, whatever, and can send that identifier back at a later date with a REQUEST (aka INVOKE) message to execute the callback by that name.

For clarity: the protocol is the format of the serialized strings passed between plugins. The implementation is anything that occurs after deserialization or before serialization. The implementation itself is irrelevant to the protocol in this design.

David Haley said:
Quote:
It just seems completely incorrect, wrong, and unintuitive to serialize the same table twice, resulting in an incorrect model after deserialization. It would would actually add to the user's burden as well as the code if I were to add this restriction.

It only adds to the user's burden if they actually care about the feature. You've made a very strong assumption that these complex tables are utterly necessary to plugin development. Can you give an example of real-world data that requires complex tables?

Mmm... I don't think they're utterly necessary. I think the point I'm trying to make now is that it's more work and burden to limit this. As for a real-world example, I can't really say I have any. I know there are situations when you want it for ease of use, but I haven't done anything quite so complex in Lua yet. I'm just very worried/irked by having a subtable (and all its own subtables) suddenly duplicated. That would be a real surprise to the user, and a subtle one too.

David Haley said:
Quote:
The difference between table keys (which I don't want) and table 'graphs' (which I do want) is that in both cases, changing it in the opposite direction would add more code and an extra burden on the user.

What is the criterion for "wanting" something?

If it makes sense, if it's clearly useful/desirable, and especially if the alternative fails one of the first previous tests. Table keys arguable fails both tests, IMO, though I can only say it doesn't make sense in the context of PPI for sure (I've explained previously why). Table graphs makes sense, although it might not be clearly useful, but the alternative makes absolutely no sense to me and is absolutely not useful or desirable.

David Haley said:
Quote:
The double-storage thing is an implementation detail specific to Lua so that I can get at the private data from within __index, based on the PPI table being accessed. It's not that confusing once you understand that.

Nothing, really, is confusing once you understand it. So this isn't really a justification for complexity.

Implementation detail, not core design issue. I can change this anytime.

David Haley said:
Quote:
I could remove the caching functionality for tables, but that would add to the complexity of the serialization routines.

How can removing code and features increase complexity?

It adds to the code because I need the cache for other reasons, which I have explained.

David Haley said:
Quote:
If you really, really wanted me to, I could change it so that it would use its real name out of the PPI table. But that would, again, create more complexity in the code just to make it look nice.

It would only adds complexity because you're trying to shoehorn it into an already complex solution; my suggestion was that the simpler solution should be the solution.

By the way, the point here is not in any fashion to make things "look nice". That is irrelevant.

The simpler solution is more complex precisely because you're trying to shoehorn things into ideals. I need to re-read what your simpler solution entailed, but from my PoV it's more work than necessary.

David Haley said:
Quote:
ACCESS and REQUEST (the latter of which should really be called INVOKE)

Then call it invoke; this is still the design phase of the protocol, and we already have irreparable backwards-compatibility brokenness to work around? ;-)

Heh, okay. ;)

David Haley said:
Quote:
Everything I've written in PPI has a distinct purpose and reasoning. It's probably not obvious to everyone reading it, that's why we're all here right now. <etc.>

I don't disbelieve that you have thought this through. But I do think that you have (with the proof-in-the-pudding, as it were) not motivated your decisions clearly enough.

By the way, you mentioned looking through the code. The issue here isn't only complexity of code, it's complexity of the idea or interface.

The interface itself is as un-complex as I could make it. Load, Expose, and treat functions as normal functions. The code is also not that complex; the 'server' stuff is at the bottom (where _G is involved), the 'client' stuff is in the middle-ish, and the serialization routines are at the top.

David Haley said:
Quote:
It's easier to unload or mix/match plugins than it is modules, IMO.

I don't know anything about this particular plugin nor frankly do I really care to delve into it. My point was just that this might be using plugins in a way they might not have originally been intended to be used. Or maybe it's a fine usage. Even so, assuming this is perfectly fine, you need to motivate the requirement for complex communication in this instance, as more than just simple data types.

True, this is likely going to be largely just simple data. There is no denying the usability of functions for callbacks, though. And this is only one use case.

David Haley said:
Quote:
The problem with table keys is that those keys are the only copy in that whole environment, because all parameters are copied. As I mentioned previously, you'd have to do a pairs() loop just to get at the keys to use them, which is rather pointless - and sending a table key -back- to the source plugin would be useless, because it's yet another copy, so the source plugin wouldn't be able to access the values it's supposed to.

This is a Lua-specific restriction. Python dictionaries-as-keys don't suffer from this problem. :-)
So why does Lua drive the language-agnostic protocol but not other languages?

Fair point. Because it's not standard or guaranteed that this feature be supported in any given language, or implemented the exact same way. It would be confusing.

David Haley said:
Quote:
The concept of a function identifier, at the protocol level, is fairly language-agnostic. Any language should be able to represent it in some manner, even if it's nothing like the Lua approach.

This is true, except that the extra complexity of generating random function identifiers is only useful to languages that have functions as anything more than names in the global namespace. And now there is inconsistency, because Lua plugins will send out identifiers as random stuff, whereas plugins in, say, VBScript will send out identifiers as actual names.

It's not random, though. You just maintain an integer and increment it every time you want a new identifier. Even VBScript is capable of this. I said it was possible that identifiers could be strings, but I never said we should mix and match these formats.
USA #87
Quote:
f:1. This is a serialized function, if you want to call it that. 'f' just denotes that this is a function-type parameter. '1' is the PPI identifier of the function. Any language can save that identifier somewhere, wrap a shim around it, whatever, and can send that identifier back at a later date with a REQUEST (aka INVOKE) message to execute the callback by that name.

For clarity: the protocol is the format of the serialized strings passed between plugins. The implementation is anything that occurs after deserialization or before serialization. The implementation itself is irrelevant to the protocol in this design.

I guess that I'm not convinced that the implementation here is so irrelevant. If there is any notion of a "PPI Identifier" for a function, then the protocol must specify what such an identifier is and what one is expected to do with it.

You can completely decouple the "PPI Identifier" from the language's identifier, but now you have created the necessity for a translation layer in between the two. So yes, your implementation makes sense in the context of requiring the translation layer. My argument is that I don't really see the benefits of this in the first place; it's yet another layer meaning yet another thing to understand, maintain, potentially bug-fix over time, etc. I see no tangible advantages to this other than being able to send unnamed functions over the API. (If security is really an issue, the plugin can restrict access to certain symbols by having a list of symbols that should not be resolved over the PPI link.)

Quote:
Mmm... I don't think they're utterly necessary. I think the point I'm trying to make now is that it's more work and burden to limit this.

I don't really understand why it's more work to limit this. It's acceptable for the API to say that behavior is undefined in the case of cyclical tables.

Quote:
As for a real-world example, I can't really say I have any. I know there are situations when you want it for ease of use, but I haven't done anything quite so complex in Lua yet.

I must be missing something here :P You say that you don't have an example but you know of situations?
If I may say, the last bit is particularly telling: you haven't done something so complex in Lua yet, but you're already baking in potential for doing something even though you don't really know what you'll use it for yet?

I don't think implementation needs to be frozen for a protocol: it's perfectly fine to allow it to grow later on if complexity emerges that must be dealt with. But I don't think that the first version should try to be a kitchen sink: it should address the problems we know we have now.

Quote:
I'm just very worried/irked by having a subtable (and all its own subtables) suddenly duplicated. That would be a real surprise to the user, and a subtle one too.

That's assuming you don't say up-front in the API documentation that only simple tables are supported, not cyclical or repetitive tables.

Furthermore, do you have an example where:
(1) you would send a table with repeated subtables,
and furthermore,
(2) the identity relationship actually matters between these subtables?

I.e., why does this duplication really matter? (It might be sending more data than it technically has to, but so what?)

Quote:
If it makes sense, if it's clearly useful/desirable, and especially if the alternative fails one of the first previous tests. Table keys arguable fails both tests, IMO, though I can only say it doesn't make sense in the context of PPI for sure (I've explained previously why).

It seems perfectly reasonable to me, if we're already discussing complex structures like cyclical tables, to want to send a mapping from coordinate to value. The coordinate would be a <x,y> pair, i.e. a table. In fact, such mappings are very easy in Python and probably Ruby, where tables-as-key-lookups are based on logical equality, not pointer equality.

Quote:
Table graphs makes sense, although it might not be clearly useful, but the alternative makes absolutely no sense to me and is absolutely not useful or desirable.

I'm not sure what alternative you're speaking of. The alternative I propose is to just not support it.

Quote:
Implementation detail, not core design issue. I can change this anytime.

When you initially discussed this piece of code, you said something about it creating PPI private space and enforcing namespaces or something along those lines. It sounds like it actually had semantic value, not just a random implementation choice.

Quote:
I need to re-read what your simpler solution entailed, but from my PoV it's more work than necessary.

This is kind of an odd statement to me, because AFAICT it's less work in the absolute and therefore could not possibly be more work than necessary when compared to a more complex solution. My proposal is that the protocol defines that functions be based around by their actual name; that's just a string and nothing needs to be done other than accept the function name from the user. Even the "fancy" solution of using debug.getinfo is less work than creating and maintaining a mapping of function value to generated function identifier.

Quote:
There is no denying the usability of functions for callbacks, though.

Indeed, but nobody has denied that. :-) What has been discussed is the usefulness of supporting arbitrary function values, which is quite different.

Quote:
Fair point. Because it's not standard or guaranteed that this feature be supported in any given language, or implemented the exact same way. It would be confusing.

Hmmmmmmmm......... :-)

Quote:
It's not random, though. You just maintain an integer and increment it every time you want a new identifier.

"Arbitrary" would have been a better word than "random", unless you consider your RNG to be just adding one. The point is that the identifier is indeed arbitrary and unrelated to the function itself. The same function value could get different identifiers depending on the order in which functions are sent around.

Quote:
Even VBScript is capable of this.

VBScript can't create a map from function value to identifier, though. In fact, VBScript doesn't even have function values.
Australia Forum Administrator #88
Worstje said:

All pre-supplied plugins tend to be useless for experienced plugin writers and new users alike.


I agree with your Worstje, probably some should be moved into an "examples" subdirectory, leaving the useful ones behind, with some better documentation.

It is interesting to look at the history of this, plugins were introduced in version 3.32 in June 2002, and Lua itself only introduced in version 3.52 in November 2004.

Thus it isn't particulary surprising that the original CallPlugin interface was not particularly friendly to Lua, as the design criteria at the time was to make a way of communicating between various plugins that was language-neutral.




At the risk of moving away from the topic a bit, I wonder whether what is being discussed here is really the main problem facing plugin developers (although quite possibly some sort of PPI could be part of the solution).

Having done some development for Aardwolf plugins I faced some interesting issues, which I will summarise and simplify here.

Say that there are two plugin developers (two different people) who want to make plugins:

  • Aard_stats.xml - shows a health bar, including what room you are in
  • Aard_minimap.xml - shows a map


Now, both plugins need to know what room you are in, and so Lasher helpfully provides this in the form of a tag, like this:


{roomname}The Aylorian Temple of Ivar


Now, normal players don't want to see this extra tag (by "normal" I mean ones not using either plugin) so there is another command added:


tag roomname on


The player (or the script) sends this tag to get the extra {roomname} line sent.

Now, both plugin writers need to have the tag turned on, so they make sure that their plugin turns it on. This isn't as easy as it looks, because you can get scenarios like this if you try to do it the moment the plugin is installed:


Welcome to Aardwolf MUD!
What be thy name, adventurer? tag roomname on
There is no player of the name "tag roomname on".


Assuming you work around all that, both plugins turn the roomname tag on (which is OK) and both omit the line "{roomname}The Aylorian Temple of Ivar" from output.

Now the fun starts. The player decides to disable the Aard_minimap.xml plugin because the map is getting in the way. The plugin realises it is being disabled and helpfully turns off the roomname tag, so it doesn't annoy the player:


tag roomname off


Uh oh! The other plugin still needs the tag, and now the Aard_stats.xml plugin stops working properly because it no longer gets the roomname tag.

One method I used to work around this was to make a third plugin (a "helper" plugin). This third plugin doesn't display any information (and thus the player is not particularly motivated to disable it). All it does is grab the {roomname} line, and make it available to other plugins. However this raises further issues:

  • How to send the data to other plugins … I used BroadcastPlugin because then I didn't need to know or care if any other plugins were installed, or wanted to know the room name. They were going to get it anyway! However because each BroadcastPlugin call identifies which plugin it came from, other plugins can either not even process the message (by not having an OnPluginBroadcast function), or if they do handle the message, check if it was from a plugin they recognised.

    This was reasonably simple and effective, and indeed the message being broadcast could be more complex than the pure inbuilt API provides for, by doing something like is done in the PPI modules, and serialising a more complex message (like, tables) into a string.
  • The system breaks if the helper plugin is not installed, or is disabled. I tried to work around this by having the dependant plugins check that the helper plugin was installed. It would be nice to automate this somehow.
  • If the helper plugin is installed, and then subsequently removed, the system may silently fail. The OnPluginListChanged callback, suggested by Twisol, may help detect this (although I don't think it detects plugins merely being disabled).
  • If all the "useful" plugins (that display stuff) are removed, and the helper plugin remains, then for efficiency purposes the helper plugin should also be disabled, to stop getting the MUD to keep asking for a roomname, which no-ones cares about. However there is no particularly easy way of handling that at present.


I can see that some form of PPI could be useful here (although I don't know that the more complex version addresses any issues the simpler one couldn't handle). For example, plugins could "register" with a helper plugin that they want something (like the room name), and when the room name arrives, all registered plugins could get it. Then when they are disabled or removed they could unregister themselves, and if no plugins were registered as wanting the room name, the helper plugin could stop requesting it.

Having said all that, the simple approach of using BroadcastPlugin is probably the easiest, and may only be slightly inefficient. Certainly some plugins may get a message they don't need, but at present Twisol's PPI script does multiple CallPlugin calls anyway, to implement a single transfer of information (the call, getting returned values, and cleaning up).

What I am presenting here is a real-life scenario showing why you may want multiple plugins, and why they might want to communicate with each other. Now the question is, to solve a real problem, as opposed to a hypothetical one (with duplicated tables etc.) would either PPI module help? Also is extra work needed (maybe adding dependency links to plugins)?
USA #89
Excellent example, Nick.

Nick Gammon said:
One method I used to work around this was to make a third plugin (a "helper" plugin). This third plugin doesn't display any information (and thus the player is not particularly motivated to disable it). All it does is grab the {roomname} line, and make it available to other plugins. However this raises further issues:

This is the approach my ATCP plugin takes as well. *nod*


Nick Gammon said:
*How to send the data to other plugins … I used BroadcastPlugin because then I didn't need to know or care if any other plugins were installed, or wanted to know the room name. They were going to get it anyway! However because each BroadcastPlugin call identifies which plugin it came from, other plugins can either not even process the message (by not having an OnPluginBroadcast function), or if they do handle the message, check if it was from a plugin they recognised.

This was reasonably simple and effective, and indeed the message being broadcast could be more complex than the pure inbuilt API provides for, by doing something like is done in the PPI modules, and serialising a more complex message (like, tables) into a string.

BroadcastPlugin does get annoying when you have multiple parameters to send, and I never liked having to check the sender's ID beforehand. It seems clearer to me if you can autometically direct these messages to specific method, too. PPI (both of ours') helps with all of these things, though since mine lets you pass functions as well, it's much easier to register asynchronous callbacks a la OnPluginBroadcast.

In simple cases like what you describe, it's less about the parameter types and more about the number of parameters for ease of use.

Nick Gammon said:
*The system breaks if the helper plugin is not installed, or is disabled. I tried to work around this by having the dependant plugins check that the helper plugin was installed. It would be nice to automate this somehow.

*nod* PPI (both of ours', I think) helps with this through the return values of PPI.Load, and OnPluginListChanged. Mine in particular (I can't remember if yours added this) stores a plugin nonce to prevent needless reloading of the PPI. If the PPI can't be loaded, you can check this by its return value and act accordingly (whether that be an error or a Note() or whatever).

Nick Gammon said:
*If the helper plugin is installed, and then subsequently removed, the system may silently fail. The OnPluginListChanged callback, suggested by Twisol, may help detect this (although I don't think it detects plugins merely being disabled).

I checked, and it does detect plugins being disabled, but not enabled. Rather odd.

Nick Gammon said:
*If all the "useful" plugins (that display stuff) are removed, and the helper plugin remains, then for efficiency purposes the helper plugin should also be disabled, to stop getting the MUD to keep asking for a roomname, which no-ones cares about. However there is no particularly easy way of handling that at present.

It depends on the helper plugin (which I prefer to call a utility plugin or resource plugin). In the case of ATCP, there's no way to disable messages after you're connected, so it just keeps stripping them out and checking to see if anyone registered for them again.

In any case, the plugin should not be disabled, or else it can't easily re-enable itself. Some part of its functionality should be disabled instead, to allow it to continue operating and re-enable things as circumstances dictate.

...Heh, I wrote this before I read your next paragraph, which pretty much said the same thing. Moving on...

Nick Gammon said:
Having said all that, the simple approach of using BroadcastPlugin is probably the easiest, and may only be slightly inefficient. Certainly some plugins may get a message they don't need, but at present Twisol's PPI script does multiple CallPlugin calls anyway, to implement a single transfer of information (the call, getting returned values, and cleaning up).

I agree, it is probably simplest, but it's also rather rigid. It's like... in C++ it's like having a lot of if statements, and having to recompile the source if you add a new object/flag, versus having a more fluid system where the object/flag is registered elsewhere, and the handler class doesn't need an if-else ladder to deal with newcomers.

I really butchered the comparison, but I hope you see what I mean. And my PPI is really not that slow anyways.

Nick Gammon said:
What I am presenting here is a real-life scenario showing why you may want multiple plugins, and why they might want to communicate with each other. Now the question is, to solve a real problem, as opposed to a hypothetical one (with duplicated tables etc.) would either PPI module help? Also is extra work needed (maybe adding dependency links to plugins)?


Both would help, but mine provides asynchronous callback functionality a la OnPluginBroadcast. The table thing is not a core issue, of course, but I'm pushing it so much because the alternative simply boggles me (duplicating tables), and could also cause infinite recursion on accident.

The separation of ACCESS and INVOKE (myppi.somevalue and somevalue()) allows attributes to be accessed more simply. It's mostly syntactic sugar, because you could emulate it almost precisely by exposing a function that returns the value, rather than the value itself, but personally I find it much clearer. I'd be willing to remove this if you want me to, but there wouldn't be much gain: I would still have to query the service at some point in order to get the requested method's PPI identifier, which is what ACCESS handles currently. And I need to manage PPI identifiers for exposed methods so that you can pass and return functions, which is key to the asynchronous callbacks technique.
USA #90
Quote:
The table thing is not a core issue, of course, but I'm pushing it so much because the alternative simply boggles me (duplicating tables)

You can think of the "alternative" as either duplicating tables, or simply not supporting tables with repeated subtables. One way of looking at it makes it a "bug", the other makes it a known not-supported feature. (This is quite serious actually and not meant to be facetious.) I think you have drawn a line too strictly between what is "technically correct" and what is "good enough to be supported". (I also think the line is somewhat arbitrary because other things are "technically correct" or possible etc. but not supported, but that's another story.)

Quote:
And I need to manage PPI identifiers for exposed methods so that you can pass and return functions, which is key to the asynchronous callbacks technique.

No you don't, not in general -- you only need to do all this because of other choices you have made. If a function is nothing more than its actual name, you don't need to worry about creating, tracking and communicating all these "PPI identifiers".
USA #91
David Haley said:
Quote:
The table thing is not a core issue, of course, but I'm pushing it so much because the alternative simply boggles me (duplicating tables)

You can think of the "alternative" as either duplicating tables, or simply not supporting tables with repeated subtables. One way of looking at it makes it a "bug", the other makes it a known not-supported feature. (This is quite serious actually and not meant to be facetious.) I think you have drawn a line too strictly between what is "technically correct" and what is "good enough to be supported". (I also think the line is somewhat arbitrary because other things are "technically correct" or possible etc. but not supported, but that's another story.)

There's no point to removing the functionality, though, because the components that make it up are also used in other parts of the library. Again, I use the cache to ensure that each table gets a separate variable (for reasons I've explained previously), so this comes for free. I could modify it so it doesn't do this, but there's no real gain to it.

David Haley said:
Quote:
And I need to manage PPI identifiers for exposed methods so that you can pass and return functions, which is key to the asynchronous callbacks technique.

No you don't, not in general -- you only need to do all this because of other choices you have made. If a function is nothing more than its actual name, you don't need to worry about creating, tracking and communicating all these "PPI identifiers".

Yes, I do. I want to support methods that might not have a name - that's exactly what allows you to do this:

PPI.Expose("foo",
  function(bar)
    -- baz
  end
)


This is intuitive, especially if you'll never use that function outside of client invocations. Plus, I need the identifiers so that I can properly support functions as parameters and returns - yes, ones without names, such as an unnamed closure! That's precisely how iterator factories like ipairs() and pairs() work, no? Certainly you might not return an iterator, but the concept has very valid use.


EDIT for clarity on the example.


EDIT: Also, I'm not sure how many other languages your name-based approach would support. By working with PPI-specific identifiers, I can decouple the idea of a 'method' from the language's implementation.
Amended on Thu 21 Jan 2010 08:46 PM by Twisol
USA #92
Quote:
Yes, I do. I want to support methods that might not have a name

Why? Is the gain of supporting nameless functions really worth the cost? Who cares if that's how things like 'ipairs' work; the question is: do we need to do that?

I mean really, do we actually need to support sending around closures? I find that remarkably unlikely, and it imposes pain upon everybody. More work has been created to support something.

I think it might help if you thought about this as a "market study" rather than "can we solve technical problems". You're adding features which require more technical solutions. But I do not believe you have really asked the question: how useful are these features actually going to be in practice? (As a general hint, saying "oh, we might want it one day" is not a satisfactory answer when it creates additional complexity.)

Frankly, the example you gave of passing an unnamed callback is the kind of thing that is "cute" to do, but not cute enough to warrant extra complexity.

Really, when compared to this:
PPI.Expose("foo",
  function(bar)
    -- baz
  end
)

why is this so much more terrible:

function foo(bar)
  do_something_with_bar()
end
PPI.Expose("foo")

OK fine, you've added a single name to the global namespace; oh no. Is the point of all this cost just to avoid adding that name to the global namespace?

Perhaps the most succinct way of making my point is: yes, you think you want that, but you shouldn't want it as an API designer unless you can justify its existence with very clear use cases where a practical problem is being solved.

Note that I'm not saying that it is impossible that we want to pass around unnamed function values. What I am saying is that we have by no means established that we do want that.

Quote:
Also, I'm not sure how many other languages your name-based approach would support. By working with PPI-specific identifiers, I can decouple the idea of a 'method' from the language's implementation.

I'm not sure what you mean. All languages can identify functions by their name as they call them, although you might need an eval statement to resolve the name reference. (But if you don't have this, the simple approach falls flat on its face anyhow, let alone the more complex approach.) Not all languages can pass around function values.

I'm not sure what you're gaining by decoupling these two ideas other than supporting unnamed functions, a concept that several languages don't even have.
USA #93
We're back to one of my initial arguments (with a slight twist):

test = PPI.Load("my id")

a = function() print('1') end
PPI.Expose("a")
-- or alternatively, since you mentioned using debug.getinfo() to get the name
PPI.Expose(a)

b = test.a
a = function() print('2') end

b()


With your approach, 2 is printed out. This doesn't makes sense to me, because if we retrieve the value of 'a', that value should be our own copy of a reference to that function, just as it is with any other PPI value including tables, and just as it is in Lua itself. With my approach, the above prints out 1.


It's certainly not just 'cute'; in the case of returning an unnamed closure, it's very, very useful, especially in the context of a resource plugin.
USA #94
Quote:
With your approach, 2 is printed out. This doesn't makes sense to me, because if we retrieve the value of 'a', that value should be our own copy of a reference to that function,

And it seems perfectly reasonable to me, if not exactly what is desirable! If the function is changed, the version that people get when they call it changes, and I would certainly not want to encourage the very wonky behavior of exposing some function and then pulling the rug out from under people. Your example seems insane to me. :-) Why would you possibly want to do such a thing? Your objection seems somewhat contrived; what is the real-life scenario where this behavior causes a problem? (And that's assuming that what you think is reasonable is in fact the objectively reasonable thing on an absolute level; as seen here, I don't really think it is.)

Besides, you keep talking about Lua semantics. Lua semantics, in your example, is that the global function 'a' is replaced with the 2 version, and any future invocation of the name "a" should give you 2. Why should exposing a name over an API be any different? Isn't it reasonable that if I change my mind about what some global symbol means, everybody who uses that global symbol should get that new version?

The point I am trying to make is not that you are wrong and I am right. The point is that it's not necessarily so obvious as you seem to think, and that there isn't one Absolutely Correct Answer.

Quote:
in the case of returning an unnamed closure, it's very, very useful, especially in the context of a resource plugin.

Give me an example. Hypothetical situations are uninteresting because obviously something will be impossible given enough constraints. The interesting question is: which constraints are actually realistic?
USA #95
David Haley said:
Besides, you keep talking about Lua semantics. Lua semantics, in your example, is that the global function 'a' is replaced with the 2 version, and any future invocation of the name "a" should give you 2. Why should exposing a name over an API be any different? Isn't it reasonable that if I change my mind about what some global symbol means, everybody who uses that global symbol should get that new version?

No, that code above is the equivalent of this native code:

a = function() print('1') end
b = a
a = function() print('2') end
b()

This returns 1.

I can see your point, but I personally think we're into the realm of personal preference now. I prefer this approach because it is closer to Lua semantics.

Besides, you could always just use myppi.MethodAgain(params), which would always return 2 in the previous example.

David Haley said:
The point I am trying to make is not that you are wrong and I am right. The point is that it's not necessarily so obvious as you seem to think, and that there isn't one Absolutely Correct Answer.

That's why I think it's more about personal preference at this point; as I've shown, my version can cater to both preferences.

David Haley said:
Quote:
in the case of returning an unnamed closure, it's very, very useful, especially in the context of a resource plugin.

Give me an example. Hypothetical situations are uninteresting because obviously something will be impossible given enough constraints. The interesting question is: which constraints are actually realistic?

Alright, iterators are an admittedly good example. A plugin might store a history of messages for each of its clients, and the client could iterate over them easily by using such a closure. This is useful for contextual evaluation, for example; you might search your history and do something specific if a certain event occurred before.

EDIT: As a slightly better example, the resource plugin would store a complete history of messages, and a plugin could iterate over it even if it didn't get any of them (i.e. if it was added after the fact).
Amended on Thu 21 Jan 2010 09:36 PM by Twisol
USA #96
Quote:
No, that code above is the equivalent of this native code:

Sorry, I don't agree. It's the equivalent only because you made it be that way by your own construction. If you are registering function names and not function values, your argument doesn't hold. You are mixing Lua semantics with the PPI semantics that you have constructed.

Regardless, you still haven't explained why one would actually want to do such a thing, though. I do not think a protocol should support stuff just because it's possible, but because it is desirable.

It sounds like what you really want is a Lua-to-Lua protocol, anyhow, given how much weight you put on preferring Lua semantics. It's unclear to me why Lua dictates everything when the explicit goal is to support cross-language, cross-plugin communication. What you're suggesting is downright weird if not impossible in languages like VBscript; I make a stronger statement in that it's also rather odd in Lua to do such a thing. Why would you ever expose a name, and then change what that name means?

Quote:
EDIT: As a slightly better example, the resource plugin would store a complete history of messages, and a plugin could iterate over it even if it didn't get any of them (i.e. if it was added after the fact).

Why must this be implemented as an iterator and not simply getting a list of the messages? The recipient would then use its native iteration as opposed to having to iterate over the PPI API.

If iteration is truly critical, then implement an iteration protocol using functions that have the iteration state as their argument one way or the other.

Don't forget that Lua isn't the only language here. VBScript might not have anything resembling this iterator construct you're discussing. In fact, this might end up creating rather awkward code in VBScript!
Amended on Thu 21 Jan 2010 09:45 PM by David Haley
USA #97
Another developer and I just discussed PPI in depth after he read this topic, and I discovered a bit of a hole in your method that affects usability.

Functions defined as values within tables have no name. Hence, you can't expose them using your method with debug.getinfo. Table functions are common enough and useful enough that it's not acceptable, in my honest opinion, to lock them out.

t = {a = function() end}
tprint(debug.getinfo(t.a))


Output:
"nups"=0
"what"="Lua"
"func"=function: 03096DD8
"lastlinedefined"=1
"source"="Command line"
"currentline"=-1
"namewhat"=""
"linedefined"=1
"short_src"="[string "Command line"]"



Happily, I also have a concrete and insanely useful example for closures. Lets take my widget framework, and create a plugin that manages some widgets. Lets say you have a second plugin that wants to do something with the first plugin's button is pressed. An idea I had, which is, yes, very practical, is to Expose a method in the resource plugin that returns a table of functions that apply specifically to the requested widget. Through this interface, you could register your own callback for a button event, and do any number of other operations the resource plugin allows, through an intuitive, easy-to-understand interface. You can treat the table returned as though it was the widget itself.

This is important, by the way, because I had been trying to come up with a solution to this long before I recreated PPI. Up until this, there was no easy way to allow other plugins to interact with my visual widgets, but PPI paves the way beautifully.
USA #98
It occurs to me that I could just use tostring(func) as the PPI identifier (i.e. "function: 03096DD8"), but it would be different for every language and I think it's probably easier just to keep it standardized.
USA #99
Quote:
Hence, you can't expose them using your method with debug.getinfo.

I never said that was useful, FWIW. I only proposed it because you insisted on passing function values.

I disagree strongly that table functions are so common, but there's really no point in making sequences of contradictory assertions. The most succinct statement at this point is that I need empirical evidence to convince me, not statements of possibility or personal belief of usefulness.

Quote:
Happily, I also have a concrete and insanely useful example for closures. Lets take my widget framework, and create a plugin that manages some widgets. Lets say you have a second plugin that wants to do something with the first plugin's button is pressed. An idea I had, which is, yes, very practical, is to Expose a method in the resource plugin that returns a table of functions that apply specifically to the requested widget. Through this interface, you could register your own callback for a button event, and do any number of other operations the resource plugin allows, through an intuitive, easy-to-understand interface. You can treat the table returned as though it was the widget itself.

I have to admit that I don't really view this as a compelling argument. It smells of separating stuff because you can, not because you should.

Anyhow, you're still quite strongly Lua-centric. I'm not sure why you're trying to argue you're not. It's not like a Lua-only cross-plugin communication layer isn't useful. In fact, if you're guaranteed to only be dealing with Lua, you can remove an awful lot of restrictions. (And, you can use "standard" serialization libraries like Pluto to really serialize stuff.)
USA #100
It's not separating so much as it is collaborating. The idea behind the widget thing isn't to separate functionality from one plugin into many, but to allow entirely separate plugins to cooperate with the original plugin in question.

I will freely admit that I may be thinking from a somewhat Lua-tainted standpoint, because I am working in Lua after all. However, I still strongly maintain that the PPI protocol layer (the serialized translated data strings) is very language independent. If you like, call the function identifier a message, and the translated data the content. This is looking very similar to OnPluginBroadcast with only a slight change in terminology. The difference is that it has the potential to be friendlier and more intuitive, especially regarding the actual data being sent. And IDs/messages can be sent to separate functions rather than the same catch-all one, and to specific/single plugins rather than all of them.

The actual inner workings of a PPI port don't need to resemble anything close to this version, nor does the interface itself. As a last resort, a port could just import the string data into MUSHclient arrays, and provide an API to translate individual entries from them manually.

I know you didn't ask about this directly, but I want to mention anyways: I wrote my own translation layer for PPI because the serialize.lua module is very Lua-centric. Its output is designed to be loadstring()()'d, and it is more effort to deserialize it properly in another language than it is to translate my PPI protocol. One reason is that I use Array*() methods to do the exporting/importing, which every language has access to. Another reason is that every piece of data is paired with a type-code that shows exactly how that piece of data should be translated.

Incidentally, Pluto happens to be a third-party library which MUSHclient doesn't include by default, which defeats the purpose of including PPI with the standard distribution. At least serialize.lua would be guaranteed to be there.



On the subject of a Lua-only PPI, yes, I could do that. I would still have to tinker with functions so that they're referred to in the source plugin and not actually serialized (which would kill upvalues and such, making them useless for plugin communication) or even ignored outright. I think I probably would have stuck with my protocol anyways, though it would undoubtedly allow for any-type keys too. But that's kind of the only 'restriction' I can see you referring to, though. What others do you see?
Amended on Fri 22 Jan 2010 06:13 AM by Twisol
USA #101
David Haley said:

I disagree strongly that table functions are so common

I don't know if it's just me, but I use table functions quite often.
USA #102
I'd be interested in an easier way to talk between my scripts, but since I write entirely in Lua, I wouldn't care if multi-language capability was included or omitted.

What interests me most is the core-modules concept this would make easier. For me though, and the reason I commented, was that I would need functions inside tables.
USA #103
To be clear, I was talking not about using them in general, but needing to communicate them across plugins. Besides, these are just another form of anonymous function; there's nothing particularly different about them being in a table or not.


function a()
  return 123
end
t = { myfunc = a }
send_to_plugin("bobs_plugin", t)

voila, tables with functions inside them being sent "over the wire". I still need a use case, though.

Anyhow, what is VBScript supposed to do with a table that contains "functions"? (The notion doesn't even exist, AFAIK...)
It'd be easier to do this kind of stuff if things were Lua-only.
USA #104
That's a rather roundabout way to go, considering that many people define the function directly in the table, or create the table and then insert the function. I also gave you a clear use case, which has implications stretching far beyond the simple (yet still very useful) example involving hooking into a button click.


Don't think of them as functions. They are message IDs. VBScript can, as I suggested, import the data into MUSHclient arrays and provide a simple API to access the entries individually. It never has to manage a structure, because it's helpfully contained by the arrays. So you might have a CallPPI() method that takes the index of the message ID to invoke in the array, and the parameters (varargs if it has it, an array if not).

EDIT: I have a better idea with VBscript involving the use of structs for individual tables/entries, and supplying the API for each entry, but since I've never used the language I don't know if it's plausible (although I certainly think it is).
Amended on Fri 22 Jan 2010 07:00 PM by Twisol
USA #105
David Haley said:

To be clear, I was talking not about using them in general, but needing to communicate them across plugins. Besides, these are just another form of anonymous function; there's nothing particularly different about them being in a table or not.

Ah, I misunderstood. As long as the function is accessible, where the function was originally in a table, I'll be alright with it. And for my uses, it wouldn't matter if I recieved a copy of the function, or access to the function in the original plugin.
USA #106
David Haley said:

I still need a use case, though.


I'm not sure if this is what you mean, but this is a function I'd make accessible to additional plugins.


rift = {
  -- stuff cut out before
  display = function (spec)
    if spec then string.lower (spec) end
    spec = rift.long[spec] or spec
    AnsiNote (ansicolor (2), "Your rift contains:")
    for k, v in ipairs (rift.cats) do
      if (spec == v) or ((not spec) and rift.groups[v]) then
        AnsiNote (ansicolor (12), "\n" .. string.proper (rift.short[v] or v))
        rift.displaygroup (v)
      end -- if
    end -- for
    prompt.draw ()
    rift.min = nil
  end, -- func
  -- stuff cut out after
  }


I'd obviously change it to work more generally, but most of the function would remain how it is.
Amended on Fri 22 Jan 2010 07:18 PM by Fadedparadox
USA #107
Twisol said:
I also gave you a clear use case, which has implications stretching far beyond the simple (yet still very useful) example involving hooking into a button click.

Not really. You said (basically) "here's a case where we can stick functions inside tables". What I want is a case where it really doesn't make any sense to do otherwise. (Coming up with situations where we "can" do this-or-that is easy and uninteresting.)

Fadedparadox said:
I'm not sure if this is what you mean, but this is a function I'd make accessible to additional plugins.

Maybe it's that I don't understand the bigger picture here, but I don't see why this has to be a function inside of a table as opposed to a function invoked with the table as an argument.

To be clear here again, the point of my comments is not that it's dumb to put functions in tables or anything like that. In fact, it makes a huge amount of sense for all kinds of reasons. My issue is with what should be done at the cross-plugin interface.
USA #108
Alright. It doesn't make sense, or at least is less clear and more typing, to create a singular method for every operation you can execute on a widget and require some ID for the widget you want to be modified to be passed to each one. Strictly speaking, my use-case makes it feasible and intuitive to give object proxies to a user. This is absolutely a case when you -should- stick functions inside tables.

Some theoretical example code (only theoretical because I haven't written it yet)
dialog = plugin.GetWidget("dialog")

ok = dialog.GetChild("button_ok")
cancel = dialog.GetChild("button_cancel")

function ButtonOK()
  print("You clicked OK")
  dialog.Destroy()
end

function ButtonCancel()
  print("You clicked Cancel")
  dialog.Destroy()
end

ok.Bind("mouseup", ButtonOK)
cancel.Bind("mouseup", ButtonCancel)



David Haley said:
To be clear here again, the point of my comments is not that it's dumb to put functions in tables or anything like that. In fact, it makes a huge amount of sense for all kinds of reasons. My issue is with what should be done at the cross-plugin interface.


To me, the interface should be able to deal with whatever personal preference the user might have. Some people like to put things in tables out of habit, as a namespacing mechanism. Other times you might have a resource, and construct a proxy object with some methods, and make the resource available via an upvalue. It just makes things so much easier and clearer to have these options, rather than be restricted to non-local, non-tabled, non-closured (i.e. just vanilla global) functions.
Australia Forum Administrator #109
Fadedparadox said:

I'm not sure if this is what you mean, but this is a function I'd make accessible to additional plugins. ...


I think the whole idea of the PPI script is to make functions available to additional plugins. However what I think David and Twisol are arguing about is *passing* functions to plugins, as arguments to another function. Maybe I'm wrong about that.

As a real-world example, say you have a status-bar plugin, and on the status bar you want to show if you are in a battle or not (eg. draw the border in red). However the status-bar plugin does not know if it is in a battle. So it calls a "battle" plugin asking to be notified when battles stop or start.

eg.


function OnStartBattle ()
  DrawBorder "red"
end 

function OnStopBattle ()
  DrawBorder "black"
end 

-- call other plugin

battle_plugin.NotifyMeOfBattles (OnStartBattle, OnStopBattle)


Now the function NotifyMeOfBattles (in the other plugin) calls back into the first plugin at the start and the end of a battle. I can see the use of this.

However as I mentioned a few pages back, which seems to have been ignored, another way of doing that is simply to have the battle plugin do a BroadcastPlugin call to tell *all* plugins (whether or not they are interested) that a battle has stopped or started.

Or, using the notify paradigm, you can pass the *names* of callbacks, like this:


function OnStartBattle ()
  DrawBorder "red"
end 

function OnStopBattle ()
  DrawBorder "black"
end 

-- call other plugin

ppi.Expose ("OnStartBattle")
ppi.Expose ("OnStopBattle")


battle_plugin.NotifyMeOfBattles ("OnStartBattle", "OnStopBattle")


This second technique first exposes the callback functions as ones available to be called back into this function, and then calls them by name (effectively, passing a string rather than a function). This is much more language-neutral.

For that matter, BroadcastPlugin is language-neutral too.
USA #110
Quote:
Alright. It doesn't make sense, or at least is less clear and more typing, to create a singular method for every operation you can execute on a widget and require some ID for the widget you want to be modified to be passed to each one. Strictly speaking, my use-case makes it feasible and intuitive to give object proxies to a user. This is absolutely a case when you -should- stick functions inside tables.

Some theoretical example code (only theoretical because I haven't written it yet)

I don't understand what this is trying to show me. I'm also not at all convinced that it makes sense to separate things by plugins as opposed to normal modules.

Anyhow, this doesn't seem unreasonable to me:


function ButtonOK()
  print("You clicked OK")
  plugin.DestroyWidget("dialog")
end

plugin.BindEvent(plugin.GetWidget("dialog", "button_ok"), "mouseup", "ButtonOK")

where GetWidget is a function that takes a list of names, and uses them as a sequence of children. So GetWidget("a", "b") would be getting the child "b" of widget "a".

It would need to be understood that the return value of GetWidget is something that the plugin can use as a widget identifier. (This is the plugin's responsibility, completely separate from PPI.)

This lets us accomplish the same thing with far simpler language constructs. No need for fancy metatables, making things callable, moving functions around, blablabla. And IMHO it would be difficult to argue that the syntax is bad, either.

Quote:
To me, the interface should be able to deal with whatever personal preference the user might have.

Go write some Perl code, or worse yet PHP, for a while, and see how happy you are with the idea that everybody's preferences, that every style of writing code should be accommodated. :-)
USA #111
Again, it's not separation, it's collaboration. The core plugin can do stuff on its own, but third-party plugins can hook into it automatically and provide extra functionality.


No, that's not unreasonable, but I do find it limiting. You have to expose a method for every potential operation at the plugin level, when it makes more sense (to me) to provide a literal object that contains them for you. Nothing in PPI prevents you from doing it the way you showed, mind you, but I think mine is more natural. Everything can be accomplished with the basic building blocks, but the whole idea is to build those into something better and easier to use.

And nobody said anything about metatables. Unless you mean the PPI_meta table and __index? How else would we save the name (or in my case, get the function ID) and store it for later use?


As for Perl/PHP, yes, I've had some experience with those. This is nothing like that kind of "please everyone" thing. I'm being no more lenient (and only very slightly less lenient) than Lua itself. Lua-centric? Sure, a bit. But my lenience also makes it easier for other languages to interface with PPI with their own constructs, whatever they might be.
USA #112
Quote:
You have to expose a method for every potential operation at the plugin level, when it makes more sense (to me) to provide a literal object that contains them for you.

I find it more useful to use objective metrics, like:
(1) what is the "expressive power" of an interface (i.e., are there some things that you simply cannot do -- and this is functionality, not style)
(2) how difficult is this "expression" (number of API calls, number of characters, etc., are all proxies for this, although they should all be taken with a grain of salt)
(3) how many functions it takes under the hood to implement all the stuff ('complexity' of the library)

Arguing about what is "natural" is a never-ending discussion where we might as well ask who prefers vanilla over chocolate. :-/

Quote:
And nobody said anything about metatables. Unless you mean the PPI_meta table and __index? How else would we save the name (or in my case, get the function ID) and store it for later use?

Well, in your example, you indeed have no choice other than to set up metatables so that things like dialog.getChild("foo") will work. My example requires no metatables at all other than for "plugin", and even that can be easily removed.

Quote:
This is nothing like that kind of "please everyone" thing.

I don't understand what you meant then when you said that every user's personal preference should be handled by the interface.

Quote:
But my lenience also makes it easier for other languages to interface with PPI with their own constructs, whatever they might be.

What is your basis for making this claim? Have you looked at what cross-language communication entails, and how your solution makes things much easier than other proposals?
USA #113
Sorry, Nick, I missed this post on accident!

Nick Gammon said:
Fadedparadox said:

I'm not sure if this is what you mean, but this is a function I'd make accessible to additional plugins. ...


I think the whole idea of the PPI script is to make functions available to additional plugins. However what I think David and Twisol are arguing about is *passing* functions to plugins, as arguments to another function. Maybe I'm wrong about that.

I think it's more about passing functions in general; the pass-by-name suggestion David had unfortunately leaves out local functions, table functions, and closures. (Remember, all of these are represented by the same exact thing on the client side, it's the service that actually deals with the function call.)

Nick Gammon said:
However as I mentioned a few pages back, which seems to have been ignored, another way of doing that is simply to have the battle plugin do a BroadcastPlugin call to tell *all* plugins (whether or not they are interested) that a battle has stopped or started.

I did give my reasons for why I thought this way was better. One is that it allows you to map incoming messages directly to functions, which does away with the if-else ladders to check ID and message. Another is, of course, the inherent rich type-passing that PPI implements.

Nick Gammon said:
Or, using the notify paradigm, you can pass the *names* of callbacks, like this:


function OnStartBattle ()
  DrawBorder "red"
end 

function OnStopBattle ()
  DrawBorder "black"
end 

-- call other plugin

ppi.Expose ("OnStartBattle")
ppi.Expose ("OnStopBattle")


battle_plugin.NotifyMeOfBattles ("OnStartBattle", "OnStopBattle")


This second technique first exposes the callback functions as ones available to be called back into this function, and then calls them by name (effectively, passing a string rather than a function). This is much more language-neutral.

I don't agree with that, to be honest. As I mentioned, passing by name requires that the function be available through _G, which locks out locals, closures, and table entries. Passing identifiers, or messages if you prefer to call them that, is in fact no different than BroadcastPlugin. As BroadcastPlugin and OnPluginBroadcast can do it in any language, it should be quite possible to implement in any language. Again, they aren't functions precisely, they are operations or messages.

Nick Gammon said:
For that matter, BroadcastPlugin is language-neutral too.

Indeed - my PPI is quite similar except for the points I mentioned earlier in the post.
USA #114
David Haley said:
Quote:
You have to expose a method for every potential operation at the plugin level, when it makes more sense (to me) to provide a literal object that contains them for you.

I find it more useful to use objective metrics, like:
(1) what is the "expressive power" of an interface (i.e., are there some things that you simply cannot do -- and this is functionality, not style)
(2) how difficult is this "expression" (number of API calls, number of characters, etc., are all proxies for this, although they should all be taken with a grain of salt)
(3) how many functions it takes under the hood to implement all the stuff ('complexity' of the library)

Arguing about what is "natural" is a never-ending discussion where we might as well ask who prefers vanilla over chocolate. :-/

I go by feel and intuition, with a good helping of imaginative forethought/critical thinking. That probably says more than anything else about our disagreements.

David Haley said:
Quote:
And nobody said anything about metatables. Unless you mean the PPI_meta table and __index? How else would we save the name (or in my case, get the function ID) and store it for later use?

Well, in your example, you indeed have no choice other than to set up metatables so that things like dialog.getChild("foo") will work. My example requires no metatables at all other than for "plugin", and even that can be easily removed.

This is a good point. I could just assign the methods I want into the proxy table, which is what I was thinking of initially, but... Hmm. I don't think it's a pressing issue - it would be more the thing the plugin author has to design around - but it's something to keep in mind for sure.

David Haley said:
Quote:
This is nothing like that kind of "please everyone" thing.

I don't understand what you meant then when you said that every user's personal preference should be handled by the interface.

I meant to have emphasis on "that kind". It is similar, but I am no more lenient than the language itself. My point is that I am also not so strict that the interface becomes some kind of neutered API. (not to imply yours is, I'm just saying.)

David Haley said:
Quote:
But my lenience also makes it easier for other languages to interface with PPI with their own constructs, whatever they might be.

What is your basis for making this claim? Have you looked at what cross-language communication entails, and how your solution makes things much easier than other proposals?

Yes, I believe I have, and I've given several examples of how other languages (in particular VBScript) can easily implement a PPI interface. I've also explained that using serialize.lua makes it more difficult for another language to parse and translate. My protocol specifically uses Array*() methods to create the translated data, so the only thing the implementation really has to do is figure out what type the data should be sent as.
Amended on Fri 22 Jan 2010 08:53 PM by Twisol
USA #115
Quote:
I go by feel and intuition, with a good helping of imaginative forethought/critical thinking. That probably says more than anything else about our disagreements.

I'm going to assume that you didn't mean I don't use forethought or critical thinking. :-P

I don't mean any offense at all here, but I think it's hard to argue for core modules on a particular person's feelings; objective metrics are at least "fair". (And if you don't like the objective metrics I proposed, you are definitely encouraged to post others.) Surely you agree that in order to have any useful discussion, we must be basing our judgments on criteria we share. There's really no point in discussing the CS equivalent of vanilla vs. chocolate.

Quote:
My point is that I am also not so strict that the interface becomes some kind of neutered API. (not to imply yours is, I'm just saying.)

The best APIs are the ones that impose a very clear policy which restricts usage just enough that it's obvious how to do the very common things, and still possible to do rather unusual things. Flexibility always comes at a cost (if anything, it becomes more confusing to do things if you have to figure out which of several options is appropriate).

Quote:
Yes, I believe I have, and I've given several examples of how other languages (in particular VBScript) can easily implement a PPI interface.

But VBScript has no notion of function values, so all the arguments made about things being "natural" in Lua do not apply at all. The PPI interface would look dramatically different. This is a loss of consistency. It's not just that function calls look different in different languages; it's that the entire approach to a problem changes even though the claim is made that it's all nominally language-neutral.

Mind you, I'm quite sympathetic to the idea that a crappy language like VBScript shouldn't drive down nicer languages. But, we're talking about cross-language development here, so we're a little restricted in how much we can ignore that crappiness.
Australia Forum Administrator #116
Twisol said:


I don't agree with that, to be honest. As I mentioned, passing by name requires that the function be available through _G, which locks out locals, closures, and table entries.


Well the function in _G can be a closure. I used that idea a bit in the module that lets you move windows around. Just to explain what I mean, this is the general idea:


function make_closure (f, colour)

return function ()
         f (colour)
       end -- inner function

end -- make_closure 


OnStartBattle = make_closure (DrawBorder, "red")
OnStopBattle = make_closure (DrawBorder, "black")

ppi.Expose ("OnStartBattle")
ppi.Expose ("OnStopBattle")


The function make_closure actually creates two closures (functions closed with respect to their local variables), even though they are the same core function.

This effectively gives us a OnStartBattle and OnStopBattle functions which "remember" which colour to draw the border in. (I admit these are both in the global namespace in this case).

In any case, assuming you want to be able to call unnamed functions, the PPI module I proposed as an alternative supports that, as in:


-- Or, for anonymous functions:
ppi.Expose ("DoSomethingElse", function () print "hello" end)



Amended on Fri 22 Jan 2010 09:05 PM by Nick Gammon
USA #117
True, but you might want to keep a list of closures (i.e. in a table), and you can't really give each one an entry in _G too. I mean, you could, but it really litters the global space, which is why you (okay, me and fadedparadox at least) generally use tables/arrays for namespacing.


True, your version does allow it because it works by key rather than by function name - you're actually storing the value, just like mine. Mine makes everything pass through PPI_REQUEST (yes, I need to change that to INVOKE) instead, so that you only have three global PPI functions (access, invoke, cleanup) rather than as many as the number of functions you exposed. However, I still think your version is somewhat less polished, no offense intended. Mine works in pretty much the same way (barring serialize.lua, and I explained why I went my own way there), except I've generalized the concept (using the same building blocks) to allow access to non-function values through the PPI. I also don't think serialize.lua will handle function parameters properly, so you can't pass function callbacks as you can in mine. Please correct me if I'm wrong!
Amended on Fri 22 Jan 2010 09:19 PM by Twisol
Australia Forum Administrator #118
serialize.lua doesn't handle functions at all.

[EDIT] I doubt you would want to.
Amended on Fri 22 Jan 2010 09:28 PM by Nick Gammon
USA #119
Nick Gammon said:

serialize.lua doesn't handle functions at all.

[EDIT] I doubt you would want to.


That's kind of what I meant. You have to pass a reference to the function, just like the PPI-exposed functions do, so to speak - each of yours is stored and kept with an associated identifier. If you want to pass a function between plugins, and have it do what you expect, you have to pass an identifier or use one it already has (the 'name' doesn't cut it because it leaves out so many kinds of functions just based on where they're defined), and pass that identifier in lieu of the function itself. That's what my PPI does, to allow function passing/returning, which in turn makes it much easier to work with the asynchronous callback/notification technique.

For what it's worth, you might want to pass a closure to be used as an async-callback function, too. You'd have a specific item out of many that you want to affect when the event occurs, and make it the upvalue to your closure.
Amended on Fri 22 Jan 2010 09:37 PM by Twisol
USA #120
Getting into a bad habit of missing peoples' posts... >_<

David Haley said:
Quote:
I go by feel and intuition, with a good helping of imaginative forethought/critical thinking. That probably says more than anything else about our disagreements.

I'm going to assume that you didn't mean I don't use forethought or critical thinking. :-P

*laughs* No, I didn't mean that at all! I meant that I think that the way I think is tempered by different drives than the way you think, in particular feel and intuition. It seems like you prefer to think at a very logical level, fleshing everything out. I could be wrong, but that's the impression I'm getting?

David Haley said:
I don't mean any offense at all here, but I think it's hard to argue for core modules on a particular person's feelings; objective metrics are at least "fair". (And if you don't like the objective metrics I proposed, you are definitely encouraged to post others.) Surely you agree that in order to have any useful discussion, we must be basing our judgments on criteria we share. There's really no point in discussing the CS equivalent of vanilla vs. chocolate.

For the sake of argument, were the other modules in MUSHclient given this in-depth of a peer review? I don't mean any offense, I'm just asking!

My main metric is whether it makes it easier on the user. You can do everything with CallPlugin or BroadcastPlugin, clearly, and indeed that's what PPI does at its core. It allows users to communicate and transport data between plugins.

Some metrics I think I tend to use, although implicitly (usually part of my feel/intuition).
1) Does it make complex operations easier to use?
2) Is it clear? Is it intuitive?
3) Does it open new possibilities? (Subjective, but important IMO)
4) Does the code itself look "good"? (Subjective/circumstantial, but if it doesn't look good then it's probably not clear)

David Haley said:
Quote:
My point is that I am also not so strict that the interface becomes some kind of neutered API. (not to imply yours is, I'm just saying.)

The best APIs are the ones that impose a very clear policy which restricts usage just enough that it's obvious how to do the very common things, and still possible to do rather unusual things. Flexibility always comes at a cost (if anything, it becomes more confusing to do things if you have to figure out which of several options is appropriate).

See, everything goes through exactly the same interface. There's no "multiple ways" with PPI, unless the user him/herself has multiple possible designs. There's always multiple routes to take no matter what you're writing. I think the great thing about my PPI is that I built it to just do the basics that I came up with at the time, and in the end it can do so much more, through exactly the same interface, and everything is still clear and makes sense semantically.

Again, you just have Load, Expose, and a handful of acceptable parameter/return types, and you can do so much right there. The procedure doesn't ever change just to pass/return/call different things.

David Haley said:
Quote:
Yes, I believe I have, and I've given several examples of how other languages (in particular VBScript) can easily implement a PPI interface.

But VBScript has no notion of function values, so all the arguments made about things being "natural" in Lua do not apply at all. The PPI interface would look dramatically different. This is a loss of consistency. It's not just that function calls look different in different languages; it's that the entire approach to a problem changes even though the claim is made that it's all nominally language-neutral.

But VBscript CAN store names and use them quite simply because of its inherent limitations, even though it still has to keep a list/dictionary (which I know VBscript has, I checked) to map between names and the protocol's function identifiers. I believe that you are still assuming that a PPI in any language has to look like a PPI in Lua. This is firmly not true, and I have never assumed that to be the case at any point in PPI's development.

David Haley said:
Mind you, I'm quite sympathetic to the idea that a crappy language like VBScript shouldn't drive down nicer languages. But, we're talking about cross-language development here, so we're a little restricted in how much we can ignore that crappiness.

So far, I haven't seen anything in PPI that VBScript can't do without a little elbow grease, and this includes things that Fadedparadox brought up during a discussion I had with him (which was itself a rather interesting take on the whole PPI ordeal). If VBscript can do it, I'm sure 99% of the other MUSH-supported languages can.
Amended on Fri 22 Jan 2010 11:05 PM by Twisol
Australia Forum Administrator #121
Twisol said:

For the sake of argument, were the other modules in MUSHclient given this in-depth of a peer review? I don't mean any offense, I'm just asking!


No, I just wrote them and posted them as part of the install files.

I think I got pulled up after the event in the case of my serialization code, where some cases were not handled correctly (funny variables names, from memory).
USA #122
Twisol said:
I meant that I think that the way I think is tempered by different drives than the way you think, in particular feel and intuition. It seems like you prefer to think at a very logical level, fleshing everything out. I could be wrong, but that's the impression I'm getting?

I guess. I start with intuition, as I think everybody does. I do try to expand intuition such that it passes clear tests; of course, intuition eventually encompasses those tests implicitly. Occasionally something will just "smell funny" and it's only after some reconstruction that I can point to the exact problems with it. If anything, this whole discussion has been an interesting exercise in fleshing out thoughts to proper arguments. :)

When I go into code-review mode, especially for something that is being considered for a standard, it is true that I refuse to accept anything unless it is very clearly motivated. APIs are so easy to screw up and bloat that great care should be taken. There's a very good reason why Lua semantics are so clean compared to, say, PHP; niceties are added with extreme caution. I've said it before, but anything based on feeling or personal preference will fail to sway me; I need very clear examples of why the simpler solutions are inappropriate.

For example, when I rewrote the complicated meta-table requiring dialog code into something far simpler, you said it was no good because it was "limiting". Well, ok. But to justify that statement, you made arguments based on which particular method more appealed to your sense of naturalness. You also said that mine was "harder to use" than yours, although I admit to finding that claim rather dubious. Basically, I think you object to it because it's not as "cool" and "shiny" (not meant to be derogatory). I have no objection to coolness and shininess per se, except when it adds complexity, and surely you agree that your meta-table-requiring approach is necessarily more complex in terms of stuff happening (for instance, it requires meta-tables...). I think you have a tendency to design things to "requirements" that you state as necessary, but haven't often called into question -- or at least justified here -- some of those basic requirements.

A question worth asking yourself re: complexity isn't so much if you understand your own code now, but if you think that coming back to it in 18 months you'll find it just as easy to understand without putting time into re-understanding it.

Twisol said:
For the sake of argument, were the other modules in MUSHclient given this in-depth of a peer review? I don't mean any offense, I'm just asking!

As Nick said, no, but I think they were also rather simpler in scope. I find it quite interesting that even the relatively simple serialization library required revisions down the road.

Twisol said:
My main metric is whether it makes it easier on the user.

This is a reasonable enough metric. But it's important to understand that it's not a one-dimensional question. If you're not careful with endless additions of syntactic sugar etc., you can end up with an implementation that's such a mess that fixing bugs and adding features becomes much harder than it needs to be.

Also, sometimes, having a toolkit with fifty different kinds of hammers for very specific variations of nails can be much harder to (learn to) use than a toolkit with a two or three hammers that cover the vast majority of cases satisfactorily.

Superficial usefulness (as in, usefulness to the user) can be countered by internal complexity; it's also extremely important to make sure that things aren't any more complex than they need to be.

I think that your other metrics are interesting but slightly misleading. An API isn't usually meant to "open possibilities", for example; it's meant to solve a problem, and the possibilities that are opened are those that come about from solving the problem easily. An API (of this sort) usually isn't a big open-ended project that ends up implementing a whole paradigm unto itself.

Twisol said:
See, everything goes through exactly the same interface. There's no "multiple ways" with PPI, unless the user him/herself has multiple possible designs.

I get the impression that you're only thinking about this from the implementor's angle, not the user. You've stated explicitly that you want all these ways of passing functions around. Yes, yes, internally PPI doesn't care if the function has a name or not. But now, users have to decide how to structure their code around all these different "possible designs".

Twisol said:
I think the great thing about my PPI is that I built it to just do the basics that I came up with at the time, and in the end it can do so much more, through exactly the same interface, and everything is still clear and makes sense semantically.

I would have thought that by now it was clear that what might be the most sensible thing semantically to you is not necessarily the most sensible thing semantically to others. Also, two words: "feature creep". Doing "so much more" is not always a good thing.

Twisol said:
I believe that you are still assuming that a PPI in any language has to look like a PPI in Lua. This is firmly not true, and I have never assumed that to be the case at any point in PPI's development.

"Has to"? No, of course not... "Should"? Actually, probably yes. Do not underestimate the value of consistency. I would say that you also should not underestimate simplicity, but I would feel like I'm repeating myself to an unsympathetic ear. ;-)

Twisol said:
So far, I haven't seen anything in PPI that VBScript can't do without a little elbow grease

Ah, well, that's kind of the rub right there. The more stuff you add, the more "elbow grease" you'll need for other languages.



If I had to choose a single takeaway for you from this entire discussion, it would be to realize that there isn't always one appropriate way of doing things; that there isn't necessarily a solution that is the best of all worlds; that there are always trade-offs. Adding features is never free.
USA #123
David Haley said:
I guess. I start with intuition, as I think everybody does. I do try to expand intuition such that it passes clear tests; of course, intuition eventually encompasses those tests implicitly. Occasionally something will just "smell funny" and it's only after some reconstruction that I can point to the exact problems with it. If anything, this whole discussion has been an interesting exercise in fleshing out thoughts to proper arguments. :)


Heh, indeed. *grin*

David Haley said:
When I go into code-review mode, especially for something that is being considered for a standard, it is true that I refuse to accept anything unless it is very clearly motivated. APIs are so easy to screw up and bloat that great care should be taken. There's a very good reason why Lua semantics are so clean compared to, say, PHP; niceties are added with extreme caution. I've said it before, but anything based on feeling or personal preference will fail to sway me; I need very clear examples of why the simpler solutions are inappropriate.


I'm doing things similarly to how Lua's doing them. I'm honestly curious, isn't that good enough reason, considering this is like a language extension unto itself?

David Haley said:
For example, when I rewrote the complicated meta-table requiring dialog code into something far simpler, you said it was no good because it was "limiting". Well, ok. But to justify that statement, you made arguments based on which particular method more appealed to your sense of naturalness. You also said that mine was "harder to use" than yours, although I admit to finding that claim rather dubious. Basically, I think you object to it because it's not as "cool" and "shiny" (not meant to be derogatory). I have no objection to coolness and shininess per se, except when it adds complexity, and surely you agree that your meta-table-requiring approach is necessarily more complex in terms of stuff happening (for instance, it requires meta-tables...). I think you have a tendency to design things to "requirements" that you state as necessary, but haven't often called into question -- or at least justified here -- some of those basic requirements.

Heheh, I didn't say it was no good, I just think it was limiting in the particular case I gave. I also never said anything about implementing metatables, I pretty much dropped that example once you pointed it out.

David Haley said:
A question worth asking yourself re: complexity isn't so much if you understand your own code now, but if you think that coming back to it in 18 months you'll find it just as easy to understand without putting time into re-understanding it.

I haven't so much as peeked at the code since my last posted change; maybe I'll give it a few more days and check back. *grin*

David Haley said:
As Nick said, no, but I think they were also rather simpler in scope. I find it quite interesting that even the relatively simple serialization library required revisions down the road.

Fair enough. I think it would be reasonable to come to a compromise on what we think we need and what we can do without for now, release it, and see what the developers think. We can always fix things down the road, as with serialize.lua. Although it might be worth noting that, whatever the complexities of my version, I think it might be easier to add/remove from without breaking things when all's said and done (although I could be wrong - I've never particularly had to maintain a library before).

Two things I would like to see kept are the protocol, and relatedly my method of passing functions around. You should be able to solve the vast majority of problems that way, with enough wiggle room to not feel closed in by my design choices.

Oh, that reminds me - that's what I meant before, about catering to personal preference. I don't want to actually restrict the user, but instead give them some wiggle room, a comfort space.

Tree/graph serialization of tables could go either way, but I have no doubt that someone won't be happy if they actually want to serialize a graph, the most likely case being a doubly-linked list in my opinion. As for whether or not they're useful, I'm using a singly-linked list in my ATCP plugin to maintain the callbacks, since it makes it easy to remove one during iteration if there's an error when it's called (if the source plugin was disabled/removed). A doubly-linked list is only a step away.

Back to the protocol itself, I want to keep it because I think it's very easy for any language to translate/deserialize, since it uses Array*() methods rather than a Lua-specific serialization technique.

David Haley said:
Twisol said:
My main metric is whether it makes it easier on the user.

This is a reasonable enough metric. But it's important to understand that it's not a one-dimensional question. If you're not careful with endless additions of syntactic sugar etc., you can end up with an implementation that's such a mess that fixing bugs and adding features becomes much harder than it needs to be.

Well, I personally think that my implementation is fairly open to changes. The protocol can easily be expanded by adding another set of conditionals to the serialization methods, for example. Everything's pretty clearly separated.

David Haley said:
Also, sometimes, having a toolkit with fifty different kinds of hammers for very specific variations of nails can be much harder to (learn to) use than a toolkit with a two or three hammers that cover the vast majority of cases satisfactorily.

I'm not understanding this, though. There's only a few multi-purpose hammers, and you use them just like the other hammers you might be used to. You can access arbitrary values and expose arbitrary values. And clearly, if the value happens to be a function, there's a way to invoke it. That's all I see. What specific hammers do you see, so I can understand your point of view?

David Haley said:
Superficial usefulness (as in, usefulness to the user) can be countered by internal complexity; it's also extremely important to make sure that things aren't any more complex than they need to be.

Well, I've stated several times that I think everything in my implementation is justified (and I've explained my views on why), and furthermore I do actually like the design when it comes right down to it.

David Haley said:
I think that your other metrics are interesting but slightly misleading. An API isn't usually meant to "open possibilities", for example; it's meant to solve a problem, and the possibilities that are opened are those that come about from solving the problem easily. An API (of this sort) usually isn't a big open-ended project that ends up implementing a whole paradigm unto itself.

Sorry, that's basically what I meant. The idea is like, you only have a certain arm length, so you can only reach so far. If you simplify a problem area, lending your own reach to the mix, then that problem becomes simpler, and new ideas might come to light because they're closer within reach.

In the end, techniques such as callback functions, which were previously available but too much work to bother with, are now within easy grasp through the same syntax that you're used to. Hence, new ideas can and will come to light that use callback functions.

David Haley said:
I get the impression that you're only thinking about this from the implementor's angle, not the user. You've stated explicitly that you want all these ways of passing functions around. Yes, yes, internally PPI doesn't care if the function has a name or not. But now, users have to decide how to structure their code around all these different "possible designs".

What I mean is, even in Lua you have a variety of ways to structure your code. PPI is no different, and it adapts readily to whatever techniques are available. PPI doesn't care if the function has a name or not because at the protocol level, it's not even a function, but a message ID akin to those sent by BroadcastPlugin. The Lua PPI simply maps them to proxy functions, which provides a deeper integration into the language's semantics.

David Haley said:
Twisol said:
I think the great thing about my PPI is that I built it to just do the basics that I came up with at the time, and in the end it can do so much more, through exactly the same interface, and everything is still clear and makes sense semantically.

I would have thought that by now it was clear that what might be the most sensible thing semantically to you is not necessarily the most sensible thing semantically to others. Also, two words: "feature creep". Doing "so much more" is not always a good thing.

I never intended to do "so much more". It just happened to come out of the design I made. No, seriously - I never made that many big changes. The only game-changing additions I made were (1) allowing functions to be passed and returned (speaking from the Lua terminology; the protocol was simply expanded to include 'f:'), and (2) separating ACCESS and INVOKE into the two messages they are now, which I am very happy I did.

David Haley said:
Twisol said:
I believe that you are still assuming that a PPI in any language has to look like a PPI in Lua. This is firmly not true, and I have never assumed that to be the case at any point in PPI's development.

"Has to"? No, of course not... "Should"? Actually, probably yes. Do not underestimate the value of consistency. I would say that you also should not underestimate simplicity, but I would feel like I'm repeating myself to an unsympathetic ear. ;-)


I went into this believing that the PPI could and should be adapted to the semantics of whatever language it was ported to, to give a closer feel to what users experienced with that language are used to. This is a core design choice that you clearly don't agree with, but I'd prefer to work with something that feels like part of the language than something that feels shoehorned in to fit. I'm not trying to convince you, but this is one of the design decisions that I won't go back on.

I'm not unsympathetic... I understand what you mean, I just disagree very much on this point. I made a choice - one that I think is a good one - and I've stuck with it.

David Haley said:
Twisol said:
So far, I haven't seen anything in PPI that VBScript can't do without a little elbow grease

Ah, well, that's kind of the rub right there. The more stuff you add, the more "elbow grease" you'll need for other languages.

I'm not planning on adding anything else, now or ever. This is sufficient for everything I would need, and everything else I've considered would be far more complex than anything else I've added, or would absolutely kill the cross-language goal. Nobody says you can't use CallPlugin yourself, either.

It's not like this was a breeze to write in Lua. I did have to put in some elbow grease of my own here. The point is that PPI would have to be ported for every language that wants to use it, but PPI is the only part that needs to be re-implemented to understand the protocol. All existing PPI-using plugins don't have to change a thing, which is in stark contrast to my original LoadPPI, which used loadstring()() on a service plugin's 'Library' MUSHclient variable, requiring the service author to write the exposed methods for every supported language themselves.


David Haley said:
If I had to choose a single takeaway for you from this entire discussion, it would be to realize that there isn't always one appropriate way of doing things; that there isn't necessarily a solution that is the best of all worlds; that there are always trade-offs. Adding features is never free.

PPI has been an exercise in compromise, though it might not seem that way to anyone else. I did have to weigh everything I added, and consider whether it would have undesirable adverse effects. There are plenty of things I could have added, but did not, because of this metric. I refer you to the post when I said "this must be desirable, that must be desirable, or the alternative must not be desirable" - that's another good metric for me. I understand your points, and believe me, I take them to heart.


EDIT: I tend to speak of the protocol and the implementation in the same breath, without noting which is which. If it's ever unclear what context I'm speaking in, pester me and I'll try to me more precise!
Amended on Sat 23 Jan 2010 08:00 AM by Twisol
USA #124
Incidentally, I have to wonder what on earth you would say to me if you reviewed the original LoadPPI code. I shudder to think! *laughs*
USA #125
Quote:
I'm doing things similarly to how Lua's doing them. I'm honestly curious, isn't that good enough reason, considering this is like a language extension unto itself?

I don't really think this should be even considered a "language extension". At its most basic, it's a way of shuttling data between plugins. That is a much simpler mission statement than a full language extension that completely drops all barriers between plugins. I also think that it covers the vast majority of use cases, and certainly all the ones we've seen so far.

Quote:
I just think it was limiting in the particular case I gave.

How? Is it limiting in functionality or not as nifty?

Quote:
I also never said anything about implementing metatables, I pretty much dropped that example once you pointed it out.

Well, ok, but how will you implement something similar? If you dropped metatables, you can't do it your way with things like ok = dialog.getChild("button_ok"). But apparently the alternative I proposed is limiting. What is your ideal solution to this?

Quote:
Although it might be worth noting that, whatever the complexities of my version, I think it might be easier to add/remove from without breaking things when all's said and done (although I could be wrong - I've never particularly had to maintain a library before).

As a general statement, internals are very easy to change except insofar as they affect externals; changing the externals is generally annoying. We wouldn't want to have something released and then say "oh hey, the API is now completely different, yay".

Quote:
I have no doubt that someone won't be happy if they actually want to serialize a graph, the most likely case being a doubly-linked list in my opinion.

I also have no doubt that somebody will want to serialize a counter function implemented with upvalues, pass it over to another plugin, and then expect the upvalues to be correctly shared across both plugins. But this isn't going to happen. :-) The point: there's lots of things that won't be possible, but the question to be asked is not "what have we limited", it's "what have we limited that actually matters".

For example: why would somebody want to send a doubly-linked list around? (And why is it important to send it as a doubly-linked list, and not just a sequence of elements?)

You already have local functions with upvalues (something I think is too complex to begin with) -- the counter example above is "only a step away". :-)

Quote:
Back to the protocol itself, I want to keep it because I think it's very easy for any language to translate/deserialize, since it uses Array*() methods rather than a Lua-specific serialization technique.

I don't think your causation is correct here (is it really the "Array" methods that make it so easy, or the text format, or...?) but yes, I agree that the serialization output should be easy to deal with. I am tempted to agree that the Lua-like serialization format might be a little annoying to deal with in VBScript, and having the type encodings makes it easy. Conversely I am tempted to believe that deserializing anything more complex than a simple map from atomic elements to atomic elements will be annoying in VBScript.

Quote:
You can access arbitrary values and expose arbitrary values. And clearly, if the value happens to be a function, there's a way to invoke it. That's all I see. What specific hammers do you see, so I can understand your point of view?

It's the "arbitrary values" part that means that people have to think more about how to interface between plugins. For example, they need to decide if they should be passing functions around as part of a table, as locals, etc. They can call functions in all kinds of ways. Your initial dialog example (the one with metatables) could have just as well implemented my version (whereas my proposal cannot have done the metatable version) -- so that's a good example of having to choose between two ways of designing something.

Quote:
In the end, techniques such as callback functions, which were previously available but too much work to bother with, are now within easy grasp through the same syntax that you're used to. Hence, new ideas can and will come to light that use callback functions.

These can be accomplished without an awful lot of the fanciness you have introduced, as my example showed.

Quote:
PPI doesn't care if the function has a name or not because at the protocol level, it's not even a function, but a message ID akin to those sent by BroadcastPlugin. The Lua PPI simply maps them to proxy functions, which provides a deeper integration into the language's semantics.

I'm not disagreeing that your PPI proposal implements more stuff. I'm just not convinced that stuff is worth the complexity at the moment. It sounds like you're trying to implement full, rich, object-oriented RPC; that seems like a much bigger scope than just shuffling data around.

Quote:
I'm not trying to convince you, but this is one of the design decisions that I won't go back on.

Well, be that as it may, but you sort of have to convince people that your approach is correct (where 'people' is a set containing at least Nick).

Quote:
I'm not planning on adding anything else, now or ever. This is sufficient for everything I would need

Frankly, I don't think it's realistic at all for you to think that you have predicted all future needs. Hence my insistence on designing to known requirements, not hypothetical requirements, because we don't know if those hypothetical requirements will ever become realized, and we certainly don't know if we've covered them all.
USA #126
David Haley said:
Quote:
I have no doubt that someone won't be happy if they actually want to serialize a graph, the most likely case being a doubly-linked list in my opinion.

I also have no doubt that somebody will want to serialize a counter function implemented with upvalues, pass it over to another plugin, and then expect the upvalues to be correctly shared across both plugins. But this isn't going to happen. :-)

Just checking in briefly, but I wanted to respond to this. What do you mean, "correctly shared across both plugins"? As far as I can understand you, they are shared correctly, because functions are not copied/executed within the client plugins, but the request is forwarded to the source plugin. It's not shared across, because there is no across for the function: it's only ever in exactly one place!

Now, if you mean you do want to copy a function to other plugins, and have it refer to that plugin's own copy of a variable, that's not hard either. You can base64-encode the string.dump() of a function, pass it over PPI as a string, and have the other side base64-decode and loadstring() it. However, that's entirely limited to Lua, and it's something the service designer would have to decide on anyways.


For the record, I'm working on a VBScript version of PPI. So far, the only troubles I'm having are dealing with the language itself. I hope to be able to post a finished version sometime soon.
USA #127
I suppose that "correctly shared" wasn't the best term. "Shared as expected" might be a better term. In some cases you might expect the upvalues to be only in one space, in others you might expect the upvalues to be unique to each scripting space.

Incidentally, this is another inconsistency. Lua semantics dictate that tables are passed by reference, and yet you pass them by value. Lua semantics also dictate that tables are passed by reference, and this time you do pass them by reference. (I.e., you're sending the closure, not a copy of the closure. But you send copies of tables, not the table reference.)

The very idea of serializing string.dump results sounds like insanity to me, by the way. :P



Here's another cross-language problem you will have. When you send a table around, how are you going to know that a given table should be created as a list or a map in the target language? Lua is somewhat unique in its agnosticism between these two types. If I send a list to VBScript (or Python or Ruby or ...) I expect to be able to treat that as a list when I receive it in Lua. You will therefore have to do all kinds of tricky business to automagically guess if a table should be converted to a list or a full map, and any automagic business is dangerous.
USA #128
David Haley said:
I suppose that "correctly shared" wasn't the best term. "Shared as expected" might be a better term. In some cases you might expect the upvalues to be only in one space, in others you might expect the upvalues to be unique to each scripting space.

Or perhaps you could "be consistent" and say that upvalues are only ever in one space, which makes it seem like just another kind of closure.

David Haley said:
Incidentally, this is another inconsistency. Lua semantics dictate that tables are passed by reference, and yet you pass them by value. Lua semantics also dictate that tables are passed by reference, and this time you do pass them by reference. (I.e., you're sending the closure, not a copy of the closure. But you send copies of tables, not the table reference.)

You're quite right. I'm just treating everything as copied 'values', and using referred-to functions (method IDs at the protocol level) for the communication. I could make it work by reference for tables, if you think I should. I could probably generalize the ACCESS message concept, from just being used by the PPI table, to being used by all passed tables. Hmm... I've been considering doing something similar within the VBScript port anyways. What do you think?

(EDIT: It might be worth noting that such a solution would make our tree/graph argument irrelevent, and also mean that there's only one MUSHclient variable used for any given INVOKE.)

David Haley said:
The very idea of serializing string.dump results sounds like insanity to me, by the way. :P

Absolutely. ;) I was just pointing out that it was possible!



David Haley said:
Here's another cross-language problem you will have. When you send a table around, how are you going to know that a given table should be created as a list or a map in the target language? Lua is somewhat unique in its agnosticism between these two types. If I send a list to VBScript (or Python or Ruby or ...) I expect to be able to treat that as a list when I receive it in Lua. You will therefore have to do all kinds of tricky business to automagically guess if a table should be converted to a list or a full map, and any automagic business is dangerous.

What I'm planning on doing in VBScript is keeping two sets of arrays, one for numeric keys and the other for string keys. I'll also provide a general Access() function that checks the type of its argument and gets from the corresponding array. It's theoretical at this point, but I think it will work.

Incidentally, I've read about a 'Scripting.Dictionary' object you can create in VBscript using CreateObject. Is that built into the engine? I haven't been able to tell for sure. If it is I'll probably use it in my port.
Amended on Sun 24 Jan 2010 06:38 AM by Twisol
USA #129
Quote:
Or perhaps you could "be consistent" and say that upvalues are only ever in one space, which makes it seem like just another kind of closure.

Yes, we get consistency in one way at the cost of expectations not being met elsewhere. And all of this comes around in the first place by claiming that we do interesting, complex things.

Quote:
I could make it work by reference for tables, if you think I should. I could probably generalize the ACCESS message concept, from just being used by the PPI table, to being used by all passed tables. Hmm... I've been considering doing something similar within the VBScript port anyways. What do you think?

Honestly, I think it would be kind of disastrous. :-) If the theme of "simplify" hasn't come across enough from me, then I have been expressing myself very poorly indeed! :P

My point with the comment was simply that "consistency with Lua semantics" has been held up as an important requirement and yet it is already being violated.

Quote:
What I'm planning on doing in VBScript is keeping two sets of arrays, one for numeric keys and the other for string keys. I'll also provide a general Access() function that checks the type of its argument and gets from the corresponding array. It's theoretical at this point, but I think it will work.

I want a list back. I don't want anything else. Unless whatever proposal you have gives me a list, it's not good.

Python is easier to illustrate this with:
widgets = plugin.GetWidgetChildren("top_level_window")
for w in widgets:
  do_something_with_w()

If "widgets" is a map type, this won't do at all what you want. It must come back as something that obeys the list iteration protocol.

(Yes, if you have a map type, then widgets[1] will do what you want whether it's a list or a map. But iteration is another story.)

You could paint a similar picture in VBScript but I don't feel like looking up the syntax etc. If the end result is a Scripting.Dictionary, that's extremely cumbersome for the VBScript programmer if all they wanted was an array. Since many languages do not make any claims about the order of keys of a mapping type, you wouldn't even be able to necessarily iterate over the mapping in the same order that Lua would use.

Basically, the point is that Lua has a somewhat peculiar way of treating arrays and maps as the same thing, whereas other languages really care about the difference. I need to see how you plan on handling this translation between idioms. (My belief: it will be hard for you, and will require a lot of magic behind the scenes. My solution? Go back to that "high level semantics" stuff I was talking about where you pass something explicitly as a sequence, a mapping, or a primitive.)

Quote:
Incidentally, I've read about a 'Scripting.Dictionary' object you can create in VBscript using CreateObject. Is that built into the engine? I haven't been able to tell for sure. If it is I'll probably use it in my port.

Yes, it exists, but no, it's not in the core and you need to include it via the "reference" mechanism. I don't know what MUSHclient does with external library references for VBScript. In Outlook etc., you need to explicitly add the references in the VBScript editor.
USA #130
David Haley said:
Quote:
Or perhaps you could "be consistent" and say that upvalues are only ever in one space, which makes it seem like just another kind of closure.

Yes, we get consistency in one way at the cost of expectations not being met elsewhere. And all of this comes around in the first place by claiming that we do interesting, complex things.

What expectations? If you want to have a single upvalue per client plugin, use a closure function returned from a factory function. Just like with vanilla Lua, the service designer decides what their API does. The client doesn't have to care in any meaningful way.

David Haley said:
Quote:
I could make it work by reference for tables, if you think I should. I could probably generalize the ACCESS message concept, from just being used by the PPI table, to being used by all passed tables. Hmm... I've been considering doing something similar within the VBScript port anyways. What do you think?

Honestly, I think it would be kind of disastrous. :-) If the theme of "simplify" hasn't come across enough from me, then I have been expressing myself very poorly indeed! :P

My point with the comment was simply that "consistency with Lua semantics" has been held up as an important requirement and yet it is already being violated.

I think it's the only real violation I've seen so far, and I'd like to think I have a decent reason for it: treating the table and its structure simply as data rather than another function-like communication mechanism. (Function-like because the table could return new values depending on when you access it.)

David Haley said:
Quote:
What I'm planning on doing in VBScript is keeping two sets of arrays, one for numeric keys and the other for string keys. I'll also provide a general Access() function that checks the type of its argument and gets from the corresponding array. It's theoretical at this point, but I think it will work.

I want a list back. I don't want anything else. Unless whatever proposal you have gives me a list, it's not good.

Python is easier to illustrate this with:
widgets = plugin.GetWidgetChildren("top_level_window")
for w in widgets:
  do_something_with_w()

If "widgets" is a map type, this won't do at all what you want. It must come back as something that obeys the list iteration protocol.

(Yes, if you have a map type, then widgets[1] will do what you want whether it's a list or a map. But iteration is another story.)

You could paint a similar picture in VBScript but I don't feel like looking up the syntax etc. If the end result is a Scripting.Dictionary, that's extremely cumbersome for the VBScript programmer if all they wanted was an array. Since many languages do not make any claims about the order of keys of a mapping type, you wouldn't even be able to necessarily iterate over the mapping in the same order that Lua would use.

Basically, the point is that Lua has a somewhat peculiar way of treating arrays and maps as the same thing, whereas other languages really care about the difference. I need to see how you plan on handling this translation between idioms. (My belief: it will be hard for you, and will require a lot of magic behind the scenes. My solution? Go back to that "high level semantics" stuff I was talking about where you pass something explicitly as a sequence, a mapping, or a primitive.)

Fair point on the list thing. Another solution I'm thinking of would be to create a key/value class, and use that for the entries. I could still use the same general API, but you could also treat it as a list and iterate over it manually.

In case you think it's strange for an access through the API to give the value, but a manual access over iteration to give the pair, C++ does do this with its std::map type.

(EDIT: I don't know why I thought it could be strange; Lua, C++, and pretty much every language I know of has this behavior on iteration! *scratch*)

David Haley said:
Quote:
Incidentally, I've read about a 'Scripting.Dictionary' object you can create in VBscript using CreateObject. Is that built into the engine? I haven't been able to tell for sure. If it is I'll probably use it in my port.

Yes, it exists, but no, it's not in the core and you need to include it via the "reference" mechanism. I don't know what MUSHclient does with external library references for VBScript. In Outlook etc., you need to explicitly add the references in the VBScript editor.

I just checked in MUSHclient, it seems to work, but I'll leave it alone. Thanks for the feedback!
Amended on Sun 24 Jan 2010 07:48 PM by Twisol
USA #131
Twisol said:
Another solution I'm thinking of would be to create a key/value class, and use that for the entries. I could still use the same general API, but you could also treat it as a list and iterate over it manually.

How would this work in Python?

Twisol said:
I have a decent reason for it: treating the table and its structure simply as data rather than another function-like communication mechanism.

I dunno, you already have tables with complex self-referential structures. That seems like more than just data to me.

And hey, you know how you were talking about the difference between referencing other tables, and having subtables? This is a great example of a difference:


my_object = { ... }
my_object.configuration = get_config() -- returns a table ref


Let's say that we send this over the wire to another plugin -- heck let's even say that the other plugin is Lua.

The configuration table will be copied.

Now the original plugin changes the global configuration for the "my_object" type.

Now, when the other plugin does stuff with "my_object", it has an outdated copy of the configuration!

(If you want a concrete example, consider things that belong to a widget, and that have a link to the widget they're contained in. You certainly don't want to copy that entire widget table. Not only is it very wasteful, it can also lead to simply incorrect and confusing behavior.)


The difference between owning subtables and referring to external tables is precisely this: when you own the subtables, it is clear that the structure is self-contained. When you refer to external tables, changing that reference is changing the semantics of the structure.
You are correct that from an implementation perspective, it's really the same thing. But from a design perspective, it's entirely different. Sometimes copying tables is fully appropriate. In other cases, it's completely wrong!

The point here: you have created the illusion of passing around stuff with full support for all kinds of things. You make a lot of effort to actually do this. But you invariably fail in some cases unless you do a lot more work in preserving references across boundaries here.

What has happened is that we have preserved (kindasorta) low-level details but completely forgotten what we were trying to do with these structures. It matters immensely that that widget referred to in the content item is a reference, and not a contained, owned subtable.

Funnily enough, a simpler protocol that makes the semantics more explicit, and tries less to preserve full implementation structure, will make this kind of thing easier. It also solves the list problem we are discussing.

Twisol said:
In case you think it's strange for an access through the API to give the value, but a manual access over iteration to give the pair, C++ does do this with its std::map type.

(EDIT: I don't know why I thought it could be strange; Lua, C++, and pretty much every language I know of has this behavior on iteration! *scratch*)

Iterating over a Python dictionary gives you the keys, although you can also iterate over the keys or values or "items" (pairs) explicitly. Iterating over a list should give you the values, not the indices.

In C++, you would most certainly not want to have to treat a list as a map from index to value.




So regarding this list stuff, we have seen that we've run into a problem you hadn't foreseen (despite thinking that you'd hammered this out and covered people's needs). This is how design works: it's extremely difficult to cover all hypothetical cases, and dangerous to think that you have. To address this whole in the complex-version-of-PPI, you will have to do a whole bunch of stuff, adding yet more complexity. And there's no guarantee that this won't happen again, or even that the solution will actually behave correctly. We've just rolled a snowball over a cliff and are watching it grow. By adding features, we have required the addition of even more features to make things "work". :-/


Quote:
I just checked in MUSHclient, it seems to work, but I'll leave it alone. Thanks for the feedback!

It's possible (or even likely) that Nick has already added the external library reference, explaining why it works.
Australia Forum Administrator #132
Twisol said:

I also added a small bit to _G[request_msg] to pass the calling plugin's ID as the first parameter, because I can see very real instances when you'd want to know the caller's ID.

  -- Call method, get return values
  local returns = {func(id, unpack(params))}
  
  -- Send returns
  send_params(returns)
end



The thing that troubles me about this change is, that the caller and callee now have asymmetrical parameter lists.

As an example, with a simple function that adds two numbers:


function add (a, b)
  return a + b
end -- add


You call that like this:


print (add (2, 3)) --> 5


However by passing the id down as a first parameter the called function's argument list does not match how you call it, so you would have to code it:


function add (id, a, b)
  return a + b
end -- add


Now I think that just looks ugly. Especially when the whole idea, I thought, was to make calling functions from one plugin to another fairly transparent, so it looks like they are really in the same plugin.

And in my "add" example, it doesn't care which module wants the numbers added together.

One way around it is to provide a "caller" local variable, which the module can check if it wants to, eg.

  
  -- Call method, get return values
  local _caller = id
  local returns = {func(unpack(params))}
  
  -- Send returns
  send_params(returns)
end


Now func "sees" a variable _caller, which it can test if it really needs to know who called it, but otherwise it can ignore it, and now the parameter lists are symmetrical.
Australia Forum Administrator #133
David Haley said:


Quote:
I just checked in MUSHclient, it seems to work, but I'll leave it alone. Thanks for the feedback!

It's possible (or even likely) that Nick has already added the external library reference, explaining why it works.


I don't think I did that, can't explain why it works. :)
USA #134
Nick Gammon said:

Twisol said:

I also added a small bit to _G[request_msg] to pass the calling plugin's ID as the first parameter, because I can see very real instances when you'd want to know the caller's ID.

  -- Call method, get return values
  local returns = {func(id, unpack(params))}
  
  -- Send returns
  send_params(returns)
end



The thing that troubles me about this change is, that the caller and callee now have asymmetrical parameter lists.

As an example, with a simple function that adds two numbers:


function add (a, b)
  return a + b
end -- add


You call that like this:


print (add (2, 3)) --> 5


However by passing the id down as a first parameter the called function's argument list does not match how you call it, so you would have to code it:


function add (id, a, b)
  return a + b
end -- add


Now I think that just looks ugly. Especially when the whole idea, I thought, was to make calling functions from one plugin to another fairly transparent, so it looks like they are really in the same plugin.

And in my "add" example, it doesn't care which module wants the numbers added together.

One way around it is to provide a "caller" local variable, which the module can check if it wants to, eg.

  
  -- Call method, get return values
  local _caller = id
  local returns = {func(unpack(params))}
  
  -- Send returns
  send_params(returns)
end


Now func "sees" a variable _caller, which it can test if it really needs to know who called it, but otherwise it can ignore it, and now the parameter lists are symmetrical.



You have a point, however I'm not sure I see how func() can access _caller, given that it's a local in the method that called it. What if I added a value 'CallerID' in the global PPI table that's set to the caller's ID when a function is being executed, and is nil otherwise?


Nick Gammon said:

David Haley said:


Quote:
I just checked in MUSHclient, it seems to work, but I'll leave it alone. Thanks for the feedback!

It's possible (or even likely) that Nick has already added the external library reference, explaining why it works.


I don't think I did that, can't explain why it works. :)

I compiled the version I'm running myself, so if it doesn't work for anyone else, maybe Visual Studio 2005 did something and I didn't realize it.
Amended on Thu 28 Jan 2010 11:27 PM by Twisol
Australia Forum Administrator #135
Twisol said:

You have a point, however I'm not sure I see how func() can access _caller, given that it's a local in the method that called it. What if I added a value 'CallerID' in the global PPI table that's set to the caller's ID when a function is being executed, and is nil otherwise?


Yes, I see that now. I assumed that CallPlugin passed down the plugin ID, but it doesn't. Anyway, assuming it is important to know the ID, I think something like what you suggested would be better.
USA #136
This has taken a while because I've been busy with some other things lately (like installing Kubuntu!), but belatedly, here is an incomplete version of VBscript-PPI that I'm still working on. It supports translation of numbers, strings, booleans, and arrays/lists, but not yet methods (which won't be that hard now that I have the infrastructure in). I've never used VBscript before, either, so I'm sure I could have done some things a lot better. Still, the point is that it does work, and I've used it to communicate non-method types values between a VBscript plugin and a Lua plugin (both ways).

It should be noted that it's wrapped in XML, as VBscript has no facility to load other files. (That said, will ImportXML() work correctly with <script> correctly? I haven't had a chance to check.) There is also no support yet for PPI_INVOKE messages, since I'm still working on method support. (You can see a PPI_Method_Class type early in the file, though.)

I'm open to any and all suggestions for improving this implementation. Again, I've never before used VBscript, so I'm probably not doing things idiomatically.


ppi.vbs.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE script>

<script><![CDATA[

' used in the PPI base class
Class PPI_Error
  Public Error
End Class

' in progress
Class PPI_Method_Class
  Public Default Function Invoke(aParams)
    Note(Join(aParams))
    Invoke = array()
  End Function
End Class


Class PPI_List_Item
  Public Key
  Public Value
End Class

Class PPI_List
  Private aEntries()
  Private numEntries
  
  Private Sub Class_Initialize()
    numEntries = 0
  End Sub
  
  Public Function GetEntry(vKey)
    Set GetEntry = Nothing
    
    ' Array keys should not be compared
    If IsArray(vKey) Then
      Exit Function
    End If
    
    If numEntries < 1 Then
      Exit Function
    End If
    
    Dim i, entry
    For i = 0 To numEntries - 1
      Set entry = aEntries(i)
      If Not entry Is Nothing Then
        If (IsObject(vKey) And IsObject(entry.Key)) Then
          If vKey Is entry.Key Then
            Set GetEntry = entry
            Exit Function
          End If
        ElseIf (Not IsObject(vKey)) And (Not IsObject(entry.Key)) Then
          If entry.Key = vKey Then
            Set GetEntry = entry
            Exit Function
          End If
        End If
      End If
    Next
  End Function
  
  Public Sub SetEntry(vKey, vValue)
    ' Get the element by key
    Dim entry
    Set entry = GetEntry(vKey)
    
    ' If it doesn't exist yet, insert a new one
    If entry Is Nothing And Not IsEmpty(vValue) Then
      Set entry = New PPI_List_Item
      
      ' Check if there are any Empty elements
      Dim i
      For i = 0 To numEntries - 1
        If IsEmpty(aEntries(i)) Then
          Set aEntries(i) = entry
          Exit For
        End If
      Next
      
      ' As a last resort, resize the array and add a new entry.
      If i = numEntries Then
        ReDim Preserve aEntries(numEntries)
        Set aEntries(numEntries) = entry
        numEntries = numEntries + 1
      End If  
    End If
    
    ' If there was one, modify it and return
    If Not entry Is Nothing Then
      ' If the value is Empty, just remove the entry
      If IsEmpty(vValue) Then
        Set aEntries(vKey) = Nothing
      Else
        If IsObject(vKey) Then
          Set entry.Key = vKey
        Else
          entry.Key = vKey
        End If
        
        If IsObject(vValue) Then
          Set entry.Value = vValue
        Else
          entry.Value = vValue
        End If
      End If
    End If
  End Sub
  
  Public Property Get At(vKey)
    Dim entry
    Set entry = GetEntry(vKey)
    If Not entry Is Nothing Then
      If IsObject(entry.Value) Then
        Set At = entry.Value
      Else
        At = entry.Value
      End If
    End If
  End Property
  
  Public Property Let At(vKey, vValue)
    SetEntry vKey, vValue
  End Property
  
  Public Property Set At(vKey, vValue)
    SetEntry vKey, vValue
  End Property
  
  Public Property Get Count
    Count = numEntries
  End Property
  
  Public Function ToArray()
    Dim aList(), entry
    ReDim aList(0)
    
    Dim i
    For i = 0 To numEntries
      Set entry = GetEntry(i)
      If entry Is Nothing Then
        Exit For
      End If
      
      If IsObject(entry.Value) Then
        Set aList(i) = entry.Value
      Else
        aList(i) = entry.Value
      End If
      
      ReDim Preserve aList(i+1)
    Next
    
    If i = 0 Then
      ToArray = array()
    Else
      ReDim Preserve aList(i-1)
      ToArray = aList
    End If
  End Function
  
  Public Function ToMap()
    If numEntries < 0 Then
      ToMap = array()
    Else
      ToMap = aEntries
    End If
  End Function
  
  Public Sub MergeArray(aItems)
    Dim i
    For i = 0 To UBound(aItems)
      SetEntry i, aItems(i)
    Next
  End Sub
End Class


Class PPI_Translate_Class
  Private listCache
  
  ' Takes a PPI_List of items and serializes them.
  Private Function Serialize_Inner(listItems, listSerialized)
    ' If we already serialized it, don't do it again
    Set Serialize_Inner = listCache.GetEntry(listItems)
    If Not Serialize_Inner Is Nothing Then
      Serialize_Inner = Serialize_Inner.Value
      Exit Function
    End If
    Serialize_Inner = listCache.Count + 1
    listCache.At(listItems) = Serialize_Inner
    
    Dim ary_id
    ary_id = "PPIarray" & Serialize_Inner
    ArrayCreate ary_id
    
    ' Convert arrays to PPI_List-style maps
    If IsArray(listItems) Then
      Dim tempList
      Set tempList = New PPI_List
      tempList.MergeArray(listItems)
      Set listItems = tempList
      Set tempList = Nothing
    End If
    listItems = listItems.ToMap()
    
    ' Go over every entry
    Dim sKey, sValue, entry
    For Each entry In listItems
      sKey = Null
      
      ' Skip invalid keys
      If Not IsObject(entry.Key) Then
        sKey = entry.Key
        If TypeName(sKey) = "String" Then
          sKey = "s:" & sKey
        ElseIf IsNumeric(sKey) Then
          sKey = "n:" & CStr(sKey + 1)
        Else
          sKey = Null
        End If
      End If
      
      If Not IsNull(sKey) Then
        sValue = "z:~"
        
        ' Default to PPI-nil on invalid values
        If Not IsObject(entry.Value) Then
          sValue = entry.Value
          
          If TypeName(sValue) = "String" Then
            sValue = "s:" & sValue
          ElseIf TypeName(sValue) = "Boolean" Then
            If sValue Then
              sValue = "b:1"
            Else
              sValue = "b:0"
            End If
          ElseIf IsNumeric(sValue) Then
            sValue = "n:" & CStr(sValue)
          ElseIf IsArray(sValue) Then
            sValue = "t:" & Serialize_Inner(sValue, listSerialized)
          End If
        ElseIf TypeName(entry.Value) = "PPI_List" Then
          sValue = "t:" & Serialize_Inner(entry.Value, listSerialized)
        End If
        
        ArraySet ary_id, sKey, sValue
      End If
    Next
    
    listSerialized.At(Serialize_Inner - 1) = ArrayExport(ary_id, "|")
    ArrayDelete ary_id
  End Function
  
  Private Function Deserialize_Inner(aItems, iIndex)
    ' If we already deserialized it, don't do it again!
    Set Deserialize_Inner = listCache.GetEntry(iIndex)
    If Not Deserialize_Inner Is Nothing Then
      If IsObject(Deserialize_Inner.Value) Then
        Set Deserialize_Inner = Deserialize_Inner.Value
      Else
        Deserialize_Inner = Deserialize_Inner.Value
      End If
      Exit Function
    End If
    Deserialize_Inner = Empty
    
    ' Used to break apart each entry
    Dim sType, sValue
    
    ' Stores the deserialized results
    Dim listResults
    Set listResults = New PPI_List
    ' Cache the result list
    Set listCache.At(iIndex) = listResults
    
    ' Import the line into an array
    ArrayCreate "PPIarray" & iIndex
    ArrayImport "PPIarray" & iIndex, aItems(iIndex), "|"
    
    Dim aKeys
    aKeys = ArrayListKeys("PPIarray" & iIndex)
    If Not IsEmpty(aKeys) Then
      ' Deserialize each item one by one
      Dim key, val
      For Each key In aKeys
        ' Keep the value for later
        val = ArrayGet("PPIarray" & iIndex, key)
        
        ' Check the keytype first
        sType = Left(key, 1)
        sValue = Right(key, Len(key) - 2)
        
        If sType = "n" Then
          key = CDbl(sValue) - 1
        ElseIf sType = "s" Then
          key = sValue
        Else
          key = Null
        End If
        
        ' If the keytype is valid, check the value
        If Not IsNull(key) Then
          ' Split into type and value
          sType = Left(val, 1)
          sValue = Right(val, Len(val) - 2)
          
          ' Deserialize by type identifier
          If sType = "s" Then
            listResults.At(key) = sValue
          ElseIf sType = "n" Then
            listResults.At(key) = CDbl(sValue)
          ElseIf sType = "b" Then
            If vValue = "1" Then
              listResults.At(key) = True
            Else
              listResults.At(key) = False
            End If
          ElseIf sType = "t" Then
            Set listResults.At(key) = Deserialize_Inner(aItems, CInt(sValue) - 1)
          Else
            listResults.At(key) = Empty
          End If
        End If
      Next
    End If
    
    ArrayDelete "PPIarray" & iIndex
    
    Set Deserialize_Inner = listResults
  End Function
  
  Public Function Serialize(listItems)
    Set listCache = New PPI_List
    Dim listSerialized
    Set listSerialized = New PPI_List
    
    Serialize_Inner listItems, listSerialized
    Serialize = listSerialized.ToArray()
    
    Set listCache = Nothing
    Set listSerialized = Nothing
  End Function
  
  Public Function Deserialize(aItems)
    Set listCache = New PPI_List
    
    Set Deserialize = Deserialize_Inner(aItems, 0)
    
    Set listCache = Nothing
  End Function
  
  Public Function GetParams(sPlugin)
    Dim aParams()
    Dim length
    length = 0
    
    Dim i
    Do While Not IsEmpty(GetPluginVariable(sPlugin, "PPIparams_" & length + 1))
      ReDim Preserve aParams(length)
      aParams(length) = GetPluginVariable(sPlugin, "PPIparams_" & length + 1)
      
      length = length + 1
    Loop
    
    If length = 0 Then
      ReDim aParams(0)
      aParams(0) = ""
    End If
    
    Set GetParams = PPI.Translate.Deserialize(aParams)
  End Function
  
  Public Sub SetParams(aParams)
    aParams = PPI.Translate.Serialize(aParams)
    
    Dim i
    For i = 0 To UBound(aParams)
      SetVariable "PPIparams_" & (i + 1), "" & aParams(i)
    Next
  End Sub
End Class


Class PPI_Accessor
  Public PluginID
  Public Reloaded
  
  Public Default Function Access(sKey)
    If Not PluginSupports(PluginID, "PPI_ACCESS") = 0 Then
      Access = Empty
      Exit Function
    End If
    
    PPI.Translate.SetParams array(sKey)
    
    CallPlugin PluginID, "PPI_ACCESS", GetPluginID()
    
    Set Access = PPI.Translate.GetParams(PluginID).GetEntry(0)
    If Access Is Nothing Then
      Access = Empty
    ElseIf IsObject(Access.Value) Then
      Set Access = Access.Value
    Else
      Access = Access.Value
    End If
    
    CallPlugin PluginID, "PPI_CLEANUP", ""
  End Function
End Class


Class PPI_Base
  Private aAccessors()
  Private listExposed
  Private myTranslate
  
  Private Sub Class_Initialize()
    Set myTranslate = New PPI_Translate_Class
    Set listExposed = New PPI_List
    
    ReDim aAccessors(0)
    aAccessors(0) = Null
  End Sub
  
  
  Public Property Get Translate()
    Set Translate = myTranslate
  End Property
  
  
  Private Sub AddAccessor(sPlugin, accessor)
    Dim length
    length = UBound(aAccessors) + 1
    
    ReDim Preserve aAccessors(length)
    aAccessors(length) = array(GetPluginID(), New PPI_Accessor)
  End Sub
  
  Private Function GetAccessor(sPlugin)
    Dim entry
    entry = Null
    
    Dim i
    For i = 1 to UBound(aAccessors)
      entry = aAccessors(i)
      If entry(0) = sPlugin Then
        Set GetAccessor = entry(1)
        Exit Function
      End If
    Next
    
    Set GetAccessor = Nothing
  End Function
  
  
  ' Load the PPI interface for a given plugin
  Public Function Load(sPlugin)
    ' Make sure the plugin is accessible
    If Not IsPluginInstalled(sPlugin) Then
      Set Load = New PPI_Error
      Load.Error = "not_installed"
      Exit Function
    ElseIf Not GetPluginInfo(sPlugin, 17) Then
      Set Load = New PPI_Error
      Load.Error = "not_enabled"
      Exit Function
    End If
    
    Dim accessor
    Set accessor = GetAccessor(sPlugin)

    ' Only create a new one if we haven't made one for it before
    If accessor Is Nothing Then
      Set accessor = New PPI_Accessor
      accessor.PluginID = sPlugin
      accessor.Reloaded = True
      AddAccessor sPlugin, accessor
    End If
    
    Set Load = accessor
  End Function
  
  Public Sub Expose(sName, vValue)
    If IsObject(vValue) Then
      Set listExposed.At(sName) = vValue
    Else
      listExposed.At(sName) = vValue
    End If
  End Sub
  
  
  Public Sub PPI_ACCESS(sPlugin)
    Dim item
    Set item = PPI.Translate.GetParams(sPlugin).GetEntry(0)
    Set item = listExposed.GetEntry(item.Value)
    
    CallPlugin PluginID, "PPI_CLEANUP", ""
    
    PPI.Translate.SetParams(array(item.Value))
  End Sub
  
  Public Sub PPI_INVOKE(sPlugin)
    ' To-do
  End Sub
  
  Public Sub PPI_CLEANUP(sPlugin)
    Dim i
    i = 1
    Do Until IsEmpty(GetVariable("PPIparams_" & i))
      DeleteVariable("PPIparams_" & i)
      i = i + 1
    Loop
  End Sub
End Class

' Create the incoming PPI message receivers
Sub PPI_INVOKE(sPlugin)
  PPI.PPI_INVOKE(sPlugin)
End Sub

Sub PPI_ACCESS(sPlugin)
  PPI.PPI_ACCESS sPlugin
End Sub

Sub PPI_CLEANUP(sPlugin)
  PPI.PPI_CLEANUP(sPlugin)
End Sub 


' The global PPI object
Dim PPI
Set PPI = New PPI_Base

]]></script>
USA #137
Example usage of the above:


Dim myppi

Sub OnPluginListChanged()
  Set myppi = PPI.Load("7c08e2961c5e20e5bdbf7fc5")
  
  ' If we had an error, the PPI couldn't be loaded.
  If TypeName(myppi) = "PPI_Error" Then
    Note("The PPI couldn't be loaded: " & myppi.Error)
    Set myppi = Nothing
    Exit Sub
  ElseIf ATCP.Reloaded Then
    Note(myppi("somevar"))
    Note(Join(myppi("somearray").ToArray(), vbCrLf))
  End If
End Sub

PPI.Expose "foo", 42
PPI.Expose "bar", "test"
PPI.Expose "baz", array(1, "two", 3.5, array(10, 42))
USA #138
Nick Gammon said:

Twisol said:

You have a point, however I'm not sure I see how func() can access _caller, given that it's a local in the method that called it. What if I added a value 'CallerID' in the global PPI table that's set to the caller's ID when a function is being executed, and is nil otherwise?


Yes, I see that now. I assumed that CallPlugin passed down the plugin ID, but it doesn't. Anyway, assuming it is important to know the ID, I think something like what you suggested would be better.


Patch:

  if not func then
    return
  end
  
  -- Call method, get return values
  PPI.CallerID = id
  local returns = {func(unpack(params))}
  PPI.CallerID = nil
  
  -- Send returns
  send_params(returns)


I also added a 'CallerID = nil,' in the PPI table for completeness.
USA #139
Quote:
What if I added a value 'CallerID' in the global PPI table that's set to the caller's ID when a function is being executed, and is nil otherwise?

RIP recursion between plugins, or indeed any form of back and forth. There's a reason why programs don't use globals to track the call stack.

Isn't it fun how complex things end up causing all this extra trouble? :-)

Seriously though; several times it has been declared that we've figured things out and that the design has covered it all, and yet several times there have been small tweaks. Not to sound like a broken record, but do we really need all those extra features that create all this extra stuff to keep track of?



Anyhow, fixes to this particular problem, assuming that it is indeed a problem that needs to be solved, would be to track a stack of callers, or perhaps to run the function in a special function environment that set the caller as a "global", rather than using an actual global.
Australia Forum Administrator #140
Twisol said:

Patch:

  if not func then
    return
  end
  
  -- Call method, get return values
  PPI.CallerID = id
  local returns = {func(unpack(params))}
  PPI.CallerID = nil
  
  -- Send returns
  send_params(returns)


I also added a 'CallerID = nil,' in the PPI table for completeness.


Where is the current source of your PPI module? In the pages of this thread I lost track of it.

Anyway, how do you know the calling id in the called plugin? I realized the target of the CallPlugin doesn't know it, so how are you setting "PPI.CallerID = id" in the *called* side? Or have I misread it?
USA #141
David Haley said:

Quote:
What if I added a value 'CallerID' in the global PPI table that's set to the caller's ID when a function is being executed, and is nil otherwise?

RIP recursion between plugins, or indeed any form of back and forth. There's a reason why programs don't use globals to track the call stack.

Isn't it fun how complex things end up causing all this extra trouble? :-)

Seriously though; several times it has been declared that we've figured things out and that the design has covered it all, and yet several times there have been small tweaks. Not to sound like a broken record, but do we really need all those extra features that create all this extra stuff to keep track of?

Anyhow, fixes to this particular problem, assuming that it is indeed a problem that needs to be solved, would be to track a stack of callers, or perhaps to run the function in a special function environment that set the caller as a "global", rather than using an actual global.


Largely a good point. I've opted to take the simple approach - also the one MUSHclient takes, IIRC - and just store the current caller ID as a local when sending an INVOKE message, to be set again on return.

In send_invoke(): (previously send_request())
  -- Prepare the arguments
  send_params({func_id, ...})
  
  -- Call the plugin
  local curr_caller = PPI.CallerID
  CallPlugin(id, invoke_msg, myID)
  PPI.CallerID = curr_caller
  
  -- Gather the return values
  local returns = receive_params(id)
USA #142
Nick Gammon said:

Twisol said:

Patch:

  if not func then
    return
  end
  
  -- Call method, get return values
  PPI.CallerID = id
  local returns = {func(unpack(params))}
  PPI.CallerID = nil
  
  -- Send returns
  send_params(returns)


I also added a 'CallerID = nil,' in the PPI table for completeness.


Where is the current source of your PPI module? In the pages of this thread I lost track of it.

Anyway, how do you know the calling id in the called plugin? I realized the target of the CallPlugin doesn't know it, so how are you setting "PPI.CallerID = id" in the *called* side? Or have I misread it?


I'll repost it now. It's probably easiest for the sake of page length to put it on a pastebin, please let me know if you'd prefer I paste it here.

As for how the called plugin knows the ID, the caller sends its ID as the third parameter to CallPlugin. That's also how the called plugin knows where to look to get the parameters.

PPI-Lua v1.2.4: http://mushclient.pastebin.com/f33d74cb1
Australia Forum Administrator #143
Twisol said:

PPI-Lua v1.2.4: http://mushclient.pastebin.com/f33d74cb1


Why not use your GitHub account? I looked there and couldn't see it. If it was there I could look at the changes between versions easily.
USA #144
Nick Gammon said:

Twisol said:

PPI-Lua v1.2.4: http://mushclient.pastebin.com/f33d74cb1


Why not use your GitHub account? I looked there and couldn't see it. If it was there I could look at the changes between versions easily.

That's an incredibly good question that I have no good answer to. I'll upload it pronto.

EDIT: Currently pushing up each version one by one so we have a decent history.
Amended on Fri 29 Jan 2010 05:12 AM by Twisol
USA #145
Okay, I've pushed a full history of PPI to a new GitHub repository. The full list of commits can be found here: http://github.com/Twisol/MUSHclient-PPI/commits/master
USA #146
I've just pushed a minor revision to PPI. It maintains backward compatibility, but makes it easier to manage the loading of PPIs.

New approach:
-- (ID, on_success, on_failure)
PPI.OnLoad("7c08e2961c5e20e5bdbf7fc5", function(atcp)
  atcp.Listen("Room.Brief", OnRoomBrief)
end, function(reason)
  -- Optional callback for if it's not available.
end)

function OnPluginListChanged()
  PPI.Refresh()
end


It also helps keep your OnPlugin* functions from getting cluttered. Many libraries want to hook into OnPlugin* hooks, but don't want to lock you out from using the callback yourself. The library simply provides functions (like PPI.Refresh()) that should be called from the appropriate OnPlugin* hook.

http://github.com/Twisol/MUSHclient-PPI