This forum is a read-only archive of the Gammon Software forum (2000–2026). No new posts can be made. Search the archive.

String.find not working for regex strings in table

Posted by Natasi on Wed 23 May 2012 02:00 AM — 12 posts, 47,332 views.

Natasi #0
I have a table (example below) filled with about 700 entries, which I go through using string.find and compare to the incoming trigger. Issue is, string.find is not working with any regex in the line, it will only match up on on plain text matches.

triggerstext = {}
triggerstext["Your mouth turns up as your face (.*?) into a smile."]=function() Note("Look mom, I found you!"); end

Here is the function I parse the trigger with:

function parseAll(label, trigger, wildcard)

for _, trig in pairs (triggerstext) do
 a = string.find(trigger, _);
 if a then
   trig();
 end
end

end

So the above will only ever match on plain, not on the regex portion. I have also written this as a regex function using, but it is extremely slow when going over 700 entries in the table :

for _, trig in pairs (triggerstext) do
re = rex.new (_);
a, b, c = re:match (trigger);
if a then
trig(c); --fires function
end
end

Any help with the string.find issue would be appreciated, or if anyone knows why the regex function is so insanely slow, that would help also. My LUA is not advanced enough yet to figure this out so far.
Fiendish USA Global Moderator #1
Lua does not use regex as part of the language. Lua uses its own pattern format described here: http://www.lua.org/pil/20.2.html

You probably want to precompile your rexes instead of doing it again 700 times every line.
Natasi #2
Ahhhh I was using (.*?) and should have been using (.*)... that fixed that issue, thanks!

As for the re:match one precompile, I tried that using the below method and it was still very slow. On load of the script I would precompile all the triggers then go through them on trigger.

function preParse()
i = 1;
for _, trig in pairs (plaintext) do
_G ["re_"..i] = rex.new (_);
i= i+1;
end
end


function parseAll1(label, trigger, wildcard)
tt = 1
for _, trig in pairs (plaintext) do
a, b, c = _G["re_"..tt]:match (trigger);
if a then
trig(c);
end
tt = tt+1;
end
end


Is there a better way of writing this?
Nick Gammon Australia Forum Administrator #3
Your original one failed because the Lua regex does not recognize (.*?) syntax. This works:


triggerstext = {}
triggerstext["Your mouth turns up as your face (.*) into a smile."]=function() Note("Look mom, I found you!"); end

function parseAll(label, trigger, wildcard)

  for pattern, trig in pairs (triggerstext) do
   if string.find (trigger, pattern) then
      trig ()
      break
   end
  end

end

local start = utils.timer ()
parseAll ("foo", "Your mouth turns up as your face Nick into a smile.")
print (utils.timer () - start)



Output:


Look mom, I found you!
0.00070651416899636


So under a millisecond for detecting that. I added the "break" because once you get a match you don't need to keep trying.

Most of that time would have been processing the match, because if I send in a non-matching string it only took about 7 microseconds.
Nick Gammon Australia Forum Administrator #4
What would speed this up, too, would be to have this trigger (which does the 700 matches) be a low-priority trigger, down the list in your trigger list behind other "more likely to match" triggers. For example, a prompt. So if you get a prompt your prompt trigger fires first, which saves even testing a single item in this table.
Nick Gammon Australia Forum Administrator #5
Look up string.find in the inbuilt help to find the exact Lua regexp syntax. The Lua regexp is quite fast, but its syntax differs a bit to the one used by triggers.

This is the web version (you get the same stuff in the help):

http://www.gammon.com.au/scripts/doc.php?lua=string.find
Amended on Wed 23 May 2012 05:58 AM by Nick Gammon
Natasi #6
If my triggertext had multiple wildcards in it, how would I capture those?

In the code below, I would want to see the first wildcard output, which would be 'up', and the second, which would be 'breaks'. I did a quick version doing a, b, c = string.find() and I could get the 'up' by outputting c, but unless I added a d, e, f (etc) it wouldn't catch additional wildcards.

triggerstext = {}
triggerstext["Your mouth turns (.*) as your face (.*) into a smile."]=function() Note("Look mom, I found you!"); end

function parseAll(label, trigger, wildcard)

for pattern, trig in pairs (triggerstext) do
if string.find (trigger, pattern) then
trig ()
break
end
end

end

local start = utils.timer ()
parseAll ("foo", "Your mouth turns up as your face Nick into a smile.")
print (utils.timer () - start)
Natasi #7
Another issue I've come across, it seems (wildcard1|wildcard2) splits do not work with Lua string.find. Would it be wise to split all my triggers apart (I have about 1284 '|' grouped triggers) into their own individual lines or try to find a way to speed up the regex version of this?
Nick Gammon Australia Forum Administrator #8
Natasi said:

If my triggertext had multiple wildcards in it, how would I capture those?


Like this:


triggerstext = {}
triggerstext["Your mouth turns (.*) as your face (.*) into a smile."]
             =
             function(where, who) 
               Note("Look " .. who .. ", I found you! and your face turned " .. where); 
             end

function parseAll(label, trigger, wildcard)

  for pattern, trig in pairs (triggerstext) do
    local results = { string.match (trigger, pattern) }
    if next (results) then
      trig (unpack (results))
      break
   end
  end

end

local start = utils.timer ()
parseAll ("foo", "Your mouth turns up as your face Nick into a smile.")
print ((utils.timer () - start) )


Output:


Look Nick, I found you! and your face turned up
0.00056068558478728



Quote:

but it is extremely slow when going over 700 entries in the table :


...

Quote:

I have about 1284 '|' grouped triggers


They seem to be multiplying!

Yes, the "|" operator does not work in Lua regexps.
Natasi #9
Thanks Nick!

The 700 entries referred to this:

triggerstext["Your mouth turns (.*) as your face (.*) into a smile."]...



The 1284 refers to the multiple within:

triggerstext["Your mouth turns (.*) as your face (.*) into a smile.|I like pie.|This is another one to match"]...


So I take it I just have to split them apart then, just to be sure, the string.math/find over the soon to be almost 2000 triggers will still parse faster than the regex over 700?
Worstje Netherlands #10
Natasi, it very likely will. The reason is that the complexity is less: PCRE has tons of options but like a car, the extra weight also carries its toll. Lua has less features and is thus less complex, meaning it can generally execute equivalent patterns faster.

Also, I have experience using several thousand builtin MUSHclient triggers myself, and while it lagged things just a little it was still something I didn't need to pay much heed to. This was 4+ years ago, so I imagine performance has only increased since.
Nick Gammon Australia Forum Administrator #11
It seems like a lot, but yes you will have to live without the "|" if you are going to use Lua.

I suggest you "anchor" them, that should make it much faster:


"^Your mouth turns (.*) as your face (.*) into a smile.$"


The reason is, without the anchor, say it gets:


There is a shop here.


Now it compares the regexp "Y" (as "Your mouth") to "T" (as in "There is a shop here."). It finds no match. But without the anchor, it then moves in one character and tries again. After all the line might say: 


There is a shop here. Your mouth turns blue as your face crinkles into a smile.


It gets no match on the "h" and tries again. And again. And again.

But with the anchor (the "^" character) if it doesn't match at the start of the line it stops immediately.

If that isn't fast enough, you could split them up a bit.

For example, all the triggers starting "You" could be put into a group (separate table). Then a single test (does the line start with "You"?) could let you decide whether to test a further 100 triggers or no.