project-2501.net » programming http://project-2501.net The net is vast and infinite. Mon, 14 May 2012 07:10:10 +0000 en hourly 1 http://wordpress.org/?v=3.2.1 PCRE and newlines (in PHP) http://project-2501.net/index.php/2012/05/pcre-and-newlines-in-php/ http://project-2501.net/index.php/2012/05/pcre-and-newlines-in-php/#comments Mon, 14 May 2012 07:09:47 +0000 Matthew http://project-2501.net/?p=440 I just wasted far too much time trying to work out why a regex would work for my example test cases but not with real data. Turns out “.”, DOT_ALL, and newlines are more complicated than I realised.

My problem essentially came down to not knowing that the imap messages I was working with used CRLF and that my OS/shell/applications were silently converting text to LF when I was constructing my test cases.

nikic has a good post covering many details with PCRE and newlines

]]> http://project-2501.net/index.php/2012/05/pcre-and-newlines-in-php/feed/ 0 Roman Numerals http://project-2501.net/index.php/2011/08/roman-numerals/ http://project-2501.net/index.php/2011/08/roman-numerals/#comments Mon, 08 Aug 2011 10:00:13 +0000 Matthew http://project-2501.net/?p=363 From reddit.com/r/programingchallenges:

I googled this and I haven’t found a similar challenge, so I’d like to pose this question to you all!
Let’s say I give you a range from 1 to 2000. Within this range, find the number that yields the most characters. I asked a friend of mine and he worked out that 1888 has a lot of characters (MDCCCLXXXVIII).

Solution

import time

SYMBOLS = [
    ('M',   1000),
    ('CM',  900),
    ('D',   500),
    ('CD',  400),
    ('C',   100),
    ('XC',  90),
    ('L',   50),
    ('XL',  40),    
    ('X',   10),
    ('IX',  9),
    ('V',   5),
    ('IV',  4),
    ('I',   1)]

def roman_numeral(number):
    roman_number = [];
    for (symbol, value) in SYMBOLS:
        while value <= number:
            roman_number.append(symbol)
            number -= value
    return ''.join(roman_number);


start = time.time();

pairs = [(i, roman_numeral(i)) for i in range(1, 2000)]
pairs.sort(lambda a,b: cmp(len(a[1]), len(b[1])))

print 'Longest roman numeral for numbers 1-2000 = %d -> %s' %  \
    (pairs[-1][0], pairs[-1][1])

print 'Took: %.2fsec' % (time.time() - start,)

Output

Longest roman numeral for numbers 1-2000 = 1888 -> MDCCCLXXXVIII
Took: 0.14sec

Notes
Turns out historically there wasn’t a strict set of rules for Roman numerals, for example IV and IIII are both valid representations of the number 4. Only recent rules have added limits on the number of repeated characters and what values can be subtracted from other values. Reference.

]]> http://project-2501.net/index.php/2011/08/roman-numerals/feed/ 0 IP Address Range PHP Class http://project-2501.net/index.php/2011/07/ip-address-range-php-class/ http://project-2501.net/index.php/2011/07/ip-address-range-php-class/#comments Thu, 28 Jul 2011 12:00:10 +0000 Matthew http://project-2501.net/?p=349 I’m not sure how useful this piece of code really is, but it gave me the chance to write some funky PHP code (have a look at the next() method).

Example usage

<?php

// Single host:
foreach(new IpRange('10.10.10.10') as $ip)
{
    echo $ip . "\n"
}

// >>> 10.10.10.10


// All hosts on a private network:
foreach(new IpRange('192.168.0.1-255') as $ip)
{
    echo $ip . "\n";
}

// >>> 192.168.0.1
// >>> 192.168.0.2
// >>> ...
// >>> 192.168.0.254
// >>> 192.168.0.255


// All normal (not broadcast, or multicast) IP addresses:
foreach(new IpRange('1-232.0-255.0-255.0-255') as $k => $v)
{
    echo "$k => $v\n";
}

// >>> 0 => 1.0.0.0
// >>> 1 => 1.0.0.1
// >>> 2 => 1.0.0.2
// >>> ...
// >>> 7315795 => 1.111.161.83
// >>> 7315796 => 1.111.161.84
// >>> etc

?>

Source

<?php

    /*

    Copyright (c) 2011, Matthew Davey <matthewd@project-2501.net>
    All rights reserved.
   
    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions are met:
   
        Redistributions of source code must retain the above copyright notice,
        this list of conditions and the following disclaimer.
       
        Redistributions in binary form must reproduce the above copyright
        notice, this list of conditions and the following disclaimer in the
        documentation and/or other materials provided with the distribution.
       
    THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
    AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
    IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
    ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
    LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
    CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
    SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
    INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
    CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
    ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
    POSSIBILITY OF SUCH DAMAGE.
   
    */
   
   
    /*
     
    IpRange Class
   
    Given an IP address where each octet can either be a number between 0 and
    255, or a range i.e. 100-200.  Return an iterable object that returns all
    IP addresses in between.
   
   
    Examples
    --------
   
   
    Single host:
        foreach(new IpRange('10.10.10.10') as $ip)
        {
            echo $ip . "\n"
        }
       
        >>> 10.10.10.10
   
   
    All hosts on a private network:
        foreach(new IpRange('192.168.0.1-255') as $ip)
        {
            echo $ip . "\n";
        }
       
        >>> 192.168.0.1
        >>> 192.168.0.2
        >>> ...
        >>> 192.168.0.254
        >>> 192.168.0.255
       
       
    All normal (not broadcast, or multicast) IP addresses:
        foreach(new IpRange('1-232.0-255.0-255.0-255') as $key => $value)
        {
            echo "$key => $value\n";
        }
       
        >>> 0 => 1.0.0.0
        >>> 1 => 1.0.0.1
        >>> 2 => 1.0.0.2
        >>> ...
        >>> 7315795 => 1.111.161.83
        >>> 7315796 => 1.111.161.84
        >>> etc
   
    */

    class IpRange implements Iterator
    {
        protected $count; // Current position, used as the key
       
        protected $a; // ranges for each octet, l => low, h => high
        protected $b; //
        protected $c; // Most significant to least is aaa.bbb.ccc.ddd
        protected $d; //
       
        protected $ca; // current value of a
        protected $cb; // current value of b
        protected $cc; // current value of c
        protected $cd; // current value of d
       
        protected $isValid; // flag set when next() is call at end of ranges
   
        public function __construct($string)
        {
            if(preg_match('#^(\d+|\d+-\d+)\.(\d+|\d+-\d+)\.(\d+|\d+-\d+)\.(\d+|\d+-\d+)$#', $string, $matches) !== 1)
            {
                throw new InvalidArgumentException('Invalid format.  Each octet should either be a number between 0 and 255, or a range "40-120"');
            }
           
            // Parse each octet and find the low/high values (high === low if there is no range specified)
            foreach(array(1 => 'a', 2 => 'b', 3 => 'c', 4 => 'd') as $i => $position)
            {
                $range = $matches[$i];

                if(strpos($range, '-') !== false)
                {
                    list($low, $high) = explode('-', $range);
                }
                else
                {
                    list($low, $high) = array($range, $range);
                }
               
                // Check the IP address is at least mostly valid.  We don't need to check for <0 as our regex will reject it first.
                if($high > 255) throw new InvalidArgumentException("Invalid IP address.  The octet '$high' cannot be greater than 255");
               
                // If given a range like 255-0, flip the high/low value over
                if($low > $high) list($high, $low) = array($low, $high);
               
                $this->{$position} = array('l' => $low, 'h' => $high);
            }
           
            $this->rewind();
        }
       
        public function current()
        {
            return "{$this->ca}.{$this->cb}.{$this->cc}.{$this->cd}";
        }
       
        public function key()
        {
            return $this->count;
        }
       
        public function rewind()
        {
            $this->ca = $this->a['l'];
            $this->cb = $this->b['l'];
            $this->cc = $this->c['l'];
            $this->cd = $this->d['l'];
           
            $this->count = 0;
           
            $this->isValid = true;
        }
       
        public function valid()
        {
            return $this->isValid;
        }
       
        public function next()
        {
            $this->count++;
           
            // Least significant to most.  Null is our guard.
            foreach(array('d', 'c', 'b', 'a', null) as $position)
            {
                // Check if unable to generate the next IP address
                if($position === null)
                {
                    $this->isValid = false;
                    break;
                }
               
                // Check if the value is less than the maximum for this
                // position, if so increment the value and stop.  Otherwise set
                // this position to its lowest value, and continue onto the next
                // highest position.
                if($this->{"c$position"} < $this->{$position}['h'])
                {
                    $this->{"c$position"}++;
                    break;
                }
                else
                {
                    $this->{"c$position"} = $this->{$position}['l'];
                }
            }
        }
    }
]]> http://project-2501.net/index.php/2011/07/ip-address-range-php-class/feed/ 0 Code Smell http://project-2501.net/index.php/2011/07/code-smell/ http://project-2501.net/index.php/2011/07/code-smell/#comments Mon, 04 Jul 2011 03:26:06 +0000 Matthew http://project-2501.net/?p=319

“In computer programming, code smell is any symptom in the source code of a program that possibly indicates a deeper problem.” — Wikipedia

I found this piece of code this morning, I think it counts as something gone terribly, terribly wrong:

return displayItemDetails(this.parentNode.parentNode.parentNode.parentNode.parentNode.parentNode, 14059111);

(The code was in-line JavaScript, inserted into a onclick handler, generated in PHP)

]]> http://project-2501.net/index.php/2011/07/code-smell/feed/ 0 Sources of Holy Power Generation http://project-2501.net/index.php/2011/04/sources-of-holy-power-generation/ http://project-2501.net/index.php/2011/04/sources-of-holy-power-generation/#comments Tue, 05 Apr 2011 08:32:17 +0000 Matthew http://project-2501.net/?p=266 Note: The information present in this post is a couple of months out of date. I’ve not had the time to re-update WoL data and create new graphs, but thought it might be interesting share the data/code anyway

When choosing a Holy Paladin spec for 10 mans I was never completely sure about benefits of Tower of Radiance or Blessed Life. ToR seems like a safe choice, though the frequency that you DL/FoL your beacon seemed low. Blessed Life on the other hand was obviously a PvP talent, but it would generate free HP from some raid damage.

I set out to scrape the top 200 Paladins for each 10-N boss fights from World of Logs and record their source of HP generation. I created two graphs; one that looks at all 200 parses in general, and a second that only counts when the ability is present. e.g. Only 10 or so Paladins used CS on Cho’gall, and they generated very little HP overall, but if you just look at lose 10 parses, the average is 8. This is used as a crude way to see when certain talents are taken.

Overall

Only when abilities are present

I’ve included the python code used to generate the data for these graphs below. My general disclaimer applies (i.e. be afraid)

grab-details.py

import urllib2
import re

pages = {
    'magmaw-10-n': ['http://worldoflogs.com/rankings/players/Blackwing_Descent/Magmaw/10N/Holy_Paladin/',
                    'http://worldoflogs.com/rankings/players/Blackwing_Descent/Magmaw/10N/Holy_Paladin/?page=2',
                    'http://worldoflogs.com/rankings/players/Blackwing_Descent/Magmaw/10N/Holy_Paladin/?page=3',
                    'http://worldoflogs.com/rankings/players/Blackwing_Descent/Magmaw/10N/Holy_Paladin/?page=4',
                    'http://worldoflogs.com/rankings/players/Blackwing_Descent/Magmaw/10N/Holy_Paladin/?page=5'],
       
    'omnitron-10-n': ['http://worldoflogs.com/rankings/players/Blackwing_Descent/Omnitron_Defense_System/10N/Holy_Paladin/',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Omnitron_Defense_System/10N/Holy_Paladin/?page=2',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Omnitron_Defense_System/10N/Holy_Paladin/?page=3',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Omnitron_Defense_System/10N/Holy_Paladin/?page=4',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Omnitron_Defense_System/10N/Holy_Paladin/?page=5'],
   
    'chimaeron-10-n': ['http://worldoflogs.com/rankings/players/Blackwing_Descent/Chimaeron/10N/Holy_Paladin/',
                       'http://worldoflogs.com/rankings/players/Blackwing_Descent/Chimaeron/10N/Holy_Paladin/?page=2',
                       'http://worldoflogs.com/rankings/players/Blackwing_Descent/Chimaeron/10N/Holy_Paladin/?page=3',
                       'http://worldoflogs.com/rankings/players/Blackwing_Descent/Chimaeron/10N/Holy_Paladin/?page=4',
                       'http://worldoflogs.com/rankings/players/Blackwing_Descent/Chimaeron/10N/Holy_Paladin/?page=5'],
   
    'atramedes-10-n': ['http://worldoflogs.com/rankings/players/Blackwing_Descent/Atramedes/10N/Holy_Paladin/',
                       'http://worldoflogs.com/rankings/players/Blackwing_Descent/Atramedes/10N/Holy_Paladin/?page=2',
                       'http://worldoflogs.com/rankings/players/Blackwing_Descent/Atramedes/10N/Holy_Paladin/?page=3',
                       'http://worldoflogs.com/rankings/players/Blackwing_Descent/Atramedes/10N/Holy_Paladin/?page=4',
                       'http://worldoflogs.com/rankings/players/Blackwing_Descent/Atramedes/10N/Holy_Paladin/?page=5'],
       
    'maloriak-10-n': ['http://worldoflogs.com/rankings/players/Blackwing_Descent/Maloriak/10N/Holy_Paladin/',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Maloriak/10N/Holy_Paladin/?page=2',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Maloriak/10N/Holy_Paladin/?page=3',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Maloriak/10N/Holy_Paladin/?page=4',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Maloriak/10N/Holy_Paladin/?page=5'],
   
    'nefarian-10-n': ['http://worldoflogs.com/rankings/players/Blackwing_Descent/Nefarian/10N/Holy_Paladin/',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Nefarian/10N/Holy_Paladin/?page=2',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Nefarian/10N/Holy_Paladin/?page=3',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Nefarian/10N/Holy_Paladin/?page=4',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Nefarian/10N/Holy_Paladin/?page=5'],
       
    'halfus-10-n': ['http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Halfus_Wyrmbreaker/10N/Holy_Paladin/',
                    'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Halfus_Wyrmbreaker/10N/Holy_Paladin/?page=2',
                    'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Halfus_Wyrmbreaker/10N/Holy_Paladin/?page=3',
                    'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Halfus_Wyrmbreaker/10N/Holy_Paladin/?page=4',
                    'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Halfus_Wyrmbreaker/10N/Holy_Paladin/?page=5'],
   
    'valiona-10-n': ['http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Valiona_&_Theralion/10N/Holy_Paladin/',
                     'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Valiona_&_Theralion/10N/Holy_Paladin/?page=2',
                     'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Valiona_&_Theralion/10N/Holy_Paladin/?page=3',
                     'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Valiona_&_Theralion/10N/Holy_Paladin/?page=4',
                     'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Valiona_&_Theralion/10N/Holy_Paladin/?page=5'],
   
    'twilight-10-n': ['http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Twilight_Ascendant_Council/10N/Holy_Paladin/',
                      'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Twilight_Ascendant_Council/10N/Holy_Paladin/?page=2',
                      'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Twilight_Ascendant_Council/10N/Holy_Paladin/?page=3',
                      'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Twilight_Ascendant_Council/10N/Holy_Paladin/?page=4',
                      'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Twilight_Ascendant_Council/10N/Holy_Paladin/?page=5'],

    'chogall-10-n': ['http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Cho\'gall/10N/Holy_Paladin/',
                     'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Cho\'gall/10N/Holy_Paladin/?page=2',
                     'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Cho\'gall/10N/Holy_Paladin/?page=3',
                     'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Cho\'gall/10N/Holy_Paladin/?page=4',
                     'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Cho\'gall/10N/Holy_Paladin/?page=5'],
   
    'conclave-10-n': ['http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Conclave_of_Wind/10N/Holy_Paladin/',
                      'http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Conclave_of_Wind/10N/Holy_Paladin/?page=2',
                      'http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Conclave_of_Wind/10N/Holy_Paladin/?page=3',
                      'http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Conclave_of_Wind/10N/Holy_Paladin/?page=4',
                      'http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Conclave_of_Wind/10N/Holy_Paladin/?page=5'],
   
    'alakir-10-n': ['http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Al\'Akir/10N/Holy_Paladin/',
                    'http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Al\'Akir/10N/Holy_Paladin/?page=2',
                    'http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Al\'Akir/10N/Holy_Paladin/?page=3',
                    'http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Al\'Akir/10N/Holy_Paladin/?page=4',
                    'http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Al\'Akir/10N/Holy_Paladin/?page=5'],
   
}

def extract_player_pages(url):
    regex   = "<td><a href='(/reports/[^']+)'>([^<]+)</a></td>"
    html    = urllib2.urlopen(url).read()
    matches = re.findall(regex, html)
   
    return matches

def extract_player_details(report_url, player_name):
    # First load the healing summary page
    url   = 'http://worldoflogs.com' + report_url
    html  = urllib2.urlopen(url).read()
    match = re.search("<a href='(/reports/[^']+)'>" + player_name + "</a>", html)
   
    # now load the players healing detail page
    url   = 'http://worldoflogs.com' + match.group(1)
    html  = urllib2.urlopen(url).read()
   
    return html


for boss in pages:
    print boss
   
    for url in pages[boss]:
        reports = extract_player_pages(url)
       
        for report_details in reports:
            report_url         = report_details[0]
            report_player_name = report_details[1]
           
            print 'Extracting', report_player_name
           
            html = extract_player_details(report_url, report_player_name)
            f    = open('data/' + boss + '/' + report_player_name, 'w+')
            f.write(html)
            f.close()
           
print 'Done'

generate-report.py

import glob
import re

# Spell IDs
HOLY_SHOCK         = 20473
ETERNAL_GLORY      = 88676
TOWER_OF_RADIANCE  = 88852
BLESSED_LIFE       = 89023
PURSUIT_OF_JUSTICE = 89024
CRUSADER_STRIKE    = 35395

bosses = [
    'alakir-10-n',
    'atramedes-10-n',
    'chimaeron-10-n',
    'chogall-10-n',
    'conclave-10-n',
    'halfus-10-n',
    'magmaw-10-n',
    'maloriak-10-n',
    'nefarian-10-n',
    'omnitron-10-n',
    'twilight-10-n',
    'valiona-10-n',
]


def extract_holy_power(html):
    regex   = r"            <td class='name'><a href='/reports/[^']+' rel='spell=(\d+)' class='spell'><span [^>]+>([^<]+)</span></a>\S+\n           <td>(\d+) holy power</td>"
    matches = re.findall(regex, html)
   
    results = {}
   
    for match in matches:
        (spell_id, spell_name, count) = match
        results[int(spell_id)] = int(count)
       
    return results


def generate_holy_power_summary(boss):
    summary = {}
   
    files = glob.glob('data/' + boss + '/*')
    for file in files:
        html = open(file, 'r').read()
        results = extract_holy_power(html)
       
        for spell_id in results:
            count = results[spell_id]
            if not summary.has_key(spell_id):
                summary[spell_id] = []
            summary[spell_id].append(count)
   
    return summary


report_on = [
    {'spell_name': 'HOLY_SHOCK', 'spell_id': HOLY_SHOCK},
    {'spell_name': 'ETERNAL_GLORY', 'spell_id': ETERNAL_GLORY},
    {'spell_name': 'TOWER_OF_RADIANCE', 'spell_id': TOWER_OF_RADIANCE},
    {'spell_name': 'BLESSED_LIFE', 'spell_id': BLESSED_LIFE},
    {'spell_name': 'PURSUIT_OF_JUSTICE', 'spell_id': PURSUIT_OF_JUSTICE},
    {'spell_name': 'CRUSADER_STRIKE', 'spell_id': CRUSADER_STRIKE},
]


for boss in bosses:
    print
    print boss
   
    summary = generate_holy_power_summary(boss)
   
    if not summary.has_key(HOLY_SHOCK):
        print 'Skipping, no data'
        continue
   
    total_count = len(summary[HOLY_SHOCK])
   
    print 'Total reports', total_count
   
    if total_count == 0:
        continue
   
    for report_details in report_on:
        if not summary.has_key(report_details['spell_id']):
            print report_details['spell_name'], 0
            continue
       
        spell_count   = len(summary[report_details['spell_id']])
        total_average = sum(summary[report_details['spell_id']]) / total_count
        spell_average = sum(summary[report_details['spell_id']]) / spell_count
       
        print report_details['spell_name'], total_average
]]> http://project-2501.net/index.php/2011/04/sources-of-holy-power-generation/feed/ 0 Text Twist Bot http://project-2501.net/index.php/2011/03/text-twist-bot/ http://project-2501.net/index.php/2011/03/text-twist-bot/#comments Wed, 16 Mar 2011 06:29:28 +0000 Matthew http://project-2501.net/?p=254 Yesterday I watched someone play a bit of the browser based game Text Twist. Upon trying myself I found that I was awful, so I did what any programmer would do; cheated.

Results of a few hours hacking:

Features:

  • Scan for game area and crash and burn if it’s not detected
  • (Crudely) detect what letters are shown
  • Find all 3, 4, 5 and 6 letter combinations, and then throw them against a basic word list + simple anagram lookup table
  • Send key presses to the window to (try and) solve all possible words (and overwrite any work you were doing when the window loses focus)
  • Reliably gets all but one or two words for each puzzle, leaving you plenty of time to go crazy trying to finish it

Horrible source for the curious (it should at least help anyone wondering how to send key presses, or capture the current screen in python)

TextTwistBot.py

# Text Twist Bot
# http://games.yahoo.com/game/text-twist

import win32com.client as comclt
from time import sleep

from PIL import ImageGrab
from itertools import combinations

from Board import Board
from Pixels import Pixels
from Anagrams import Anagrams


def sorted_unique_character_groups(letters, length):
    c = list(combinations(''.join(letters), length))
    c = [''.join(e) for e in c]
    c = list(set(c))
    c.sort()
    return c


anagrams = Anagrams('ispell-enwl-3.1.20/english.all')
pixels   = Pixels(ImageGrab.grab())
board    = Board(pixels)
letters  = board.get_letters()

# letters = ['d', 'e', 'l', 'a', 'p', 'd']
possible_answers = []
lengths = [3,4,5,6]

for length in lengths:
    words = sorted_unique_character_groups(letters, length)
    for word in words:
        possible_answers.extend(anagrams.find(word))

possible_answers = list(set(possible_answers))

print possible_answers


wsh = comclt.Dispatch("WScript.Shell")
wsh.AppActivate("Windows Internet Explorer")

for word in possible_answers:
    for letter in word:
        wsh.SendKeys(letter)
        sleep(0.05)
    wsh.SendKeys("\n")

print 'Done'

Pixels.py

class Pixels:
    def __init__(self, img):
        self.img = img
        self.data = list(img.getdata())
        (self.width, self.height) = img.size
       
    def at(self, x, y):
        offset = (y * self.width) + x
        return self.data[offset]
       
    def grab_area(self, x1, y1, x2, y2):
        raw = []
        for y in range(y1, y2):
            for x in range(x1, x2):
                raw.append(self.at(x, y))
        return raw

Anagrams.py

class Anagrams:
    def __init__(self, word_list):
        self._load_word_list(word_list)
       
    def _load_word_list(self, filename):
        print 'Loading word list...',
        lines = open(filename).readlines()
        words = [line.strip().lower() for line in lines]
       
        self.lookup = {}
       
        for word in words:
            key = self._sort_letters(word)
            if key not in self.lookup:
                self.lookup[key] = []
            if word not in self.lookup[key]:
                self.lookup[key].append(word)
        print 'Done'
       
    def _sort_letters(self, word):
        letters = [l for l in word]
        letters.sort()
        return ''.join(letters)
       
    def find(self, word):
        key = self._sort_letters(word)
        if key not in self.lookup:
            return []
        results = self.lookup[key]
        #if word in results:
        #    results.remove(word)
        return results

Board.py (be afraid)

class Board:
    def __init__(self, pixels):
       
        # Color of the outside border, used to find the edges of the game
        self.border_green = (204, 255, 0)
       
        # Bounding boxes for each letter (including the circle and background)
        # Each letter is 45x45
        # I've chopped off 8pixels from each side to remove any background
        self.letter_coords = [
            ((161+8, 178+8), (161+45-8, 178+45-8)),
            ((215+8, 178+8), (215+45-8, 178+45-8)),
            ((269+8, 178+8), (269+45-8, 178+45-8)),
            ((323+8, 178+8), (323+45-8, 178+45-8)),
            ((377+8, 178+8), (377+45-8, 178+45-8)),
            ((431+8, 178+8), (431+45-8, 178+45-8)),
        ]
       
        # Pixel data for each letter
        self.letter_data = {
            # Snipped 300Kb of data
            # Yikes, didn't realize it was so big
            # If anyone actually cares for the letter data, you
            # can reconstruct it from error messages.
            #
            # Format is:
            #
            # 'a': [(r, g, b), (r, g, b)....],
            # 'b': [(r, g, b), (r, g, b)....],
        }
       
        self.pixels = pixels
        self._get_edges()
       
    def _get_edges(self):
        horz = self._find_pixels(range(0, self.pixels.width),      range(0, self.pixels.height, 100), self.border_green)
        vert = self._find_pixels(range(0, self.pixels.width, 100), range(0, self.pixels.height),      self.border_green)
       
        left   = min([x for (x,y) in horz])
        right  = max([x for (x,y) in horz])
        top    = min([y for (x,y) in vert])
        bottom = max([y for (x,y) in vert])
       
        self.left   = left
        self.right  = right
        self.top    = top
        self.bottom = bottom
       
    def _find_pixels(self, x_range, y_range, color):
        edges = []
        for y in y_range:
            for x in x_range:
                if self.pixels.at(x, y) == color:
                    edges.append((x, y))
        return edges
   
    def get_letters(self):
        letters = []
        for coords in self.letter_coords:
            x1 = coords[0][0] + self.left
            x2 = coords[1][0] + self.left
           
            y1 = coords[0][1] + self.top
            y2 = coords[1][1] + self.top
       
            data = self.pixels.grab_area(x1, y1, x2, y2)
           
            # print 'Got data for pos', coords
            found = False
            for key in self.letter_data:
                if self.letter_data[key] == data:
                    letters.append(key)
                    found = True
                    break
           
            if found == False:
                print '\'?\':', data, ','
        return letters
]]> http://project-2501.net/index.php/2011/03/text-twist-bot/feed/ 2 Intial Divine Guardian Tests http://project-2501.net/index.php/2010/02/intial-divine-guardian-tests/ http://project-2501.net/index.php/2010/02/intial-divine-guardian-tests/#comments Wed, 03 Feb 2010 08:51:50 +0000 Matthew http://project-2501.net/?p=233 I been wanting to look in the effectiveness of the 20% raid wall granted by Divine Guardian (4th tier talent in a Paladin’s protection tree). To do this I wrote a very simple Python program to read the combat log, detect with DG goes up and then record all the damage that was taken while it was present on a unit, and work out how much was mitigated. Of course, as the script was hacked up, it has all sorts of limitations:

  • Ignores Divine Sacrifice
  • Ignores any other mitigation effects (Talents / Sanc / Inspiration / etc)
  • Ignores overkills
  • Undefined behavior when used by two Paladins

Below is the results for a two nights in ICC 25, while I’m sure it’s not 100% it should be a reasonable ballpark figure.

Fight Damage Mitigated via DG
Gaseous Blight 37057
Gaseous Blight 32664
Ooze Explosion/Melee 39923
Trash before Blood Queen 21145
Blood Queen (Fear/Bloodbolt) 42212
Blood Queen (Fear/Bloodbolt) 54410
Blood Queen (Fear/Bloodbolt) 17679
Blood Queen (Fear/Bloodbolt) 29315
Blood Queen (Fear/Bloodbolt) 32029
Blood Queen (Fear/Bloodbolt) 51281
Blood Queen (Fear/Bloodbolt) 34494
Blood Queen (Fear/Bloodbolt) 14201
Blood Queen (Fear/Bloodbolt) 41417
Blood Queen (Fear/Bloodbolt) 18817
Blood Queen (Fear/Bloodbolt) 35641
Blood Queen (Fear/Bloodbolt) 49571
Blood Queen (Fear/Bloodbolt) 26427
Average 34017
]]> http://project-2501.net/index.php/2010/02/intial-divine-guardian-tests/feed/ 0 javascript; sending me insane http://project-2501.net/index.php/2010/01/javascript-sending-me-insane/ http://project-2501.net/index.php/2010/01/javascript-sending-me-insane/#comments Tue, 19 Jan 2010 03:55:38 +0000 Matthew http://project-2501.net/?p=228
>>> []
[]
>>> [] == true
false
>>> [] == false
true
>>> [] == []
false
>>> false == false
true
]]> http://project-2501.net/index.php/2010/01/javascript-sending-me-insane/feed/ 1 total mysql rows http://project-2501.net/index.php/2009/09/total-mysql-rows/ http://project-2501.net/index.php/2009/09/total-mysql-rows/#comments Wed, 09 Sep 2009 01:50:04 +0000 Matthew http://project-2501.net/?p=213 Turns out our Mysql server at work is a little bigger than I thought:

Databases 75
Tables    1,549
Rows      1,018,085,348

However over the last couple of months, we’ve only averaged 130 queries/second

Hacked up PHP to gather stats:

class MysqlCounter
{
    public function __construct($host, $username, $password)
    {
        $this->conn = mysql_connect($host, $username, $password, true);
           
        if($this->conn === false)
        {
            throw new Exception("Unable to connect to Mysql server: " . mysql_error());
        }
           
        $this->num_databases = 0;
        $this->num_tables = 0;
        $this->num_rows = 0;
    }
       
    private function query($sql)
    {
        $query = mysql_query($sql, $this->conn);
           
        if($query === false)
        {
            throw new Exception("Unable to run query: $sql\n" . mysql_error($this->conn));
        }
           
        if($query === true)
        {
            return array();
        }
           
        $rows = array();
           
        while ($row = mysql_fetch_array($query))
        {
            $rows[] = $row;
        }
           
        return $rows;
    }
       
    public function gather_stats($callback = false)
    {
        foreach($this->query("SHOW DATABASES") as $row_database)
        {
            $this->num_databases++;
               
            $this->query("USE `{$row_database['Database']}`");
               
            foreach($this->query("SHOW TABLE STATUS") as $row_table)
            {
                $this->num_tables++;
                $this->num_rows += $row_table['Rows'];
                   
                if($callback !== false)
                {
                    $args = array($row_database['Database'], $row_table['Name'], $row_table['Rows']);
                    call_user_func_array($callback, $args);
                }
            }
        }
    }
       
    public static function default_callback($database, $table, $rows)
    {
        printf("%s %s %d\n", $database, $table, $rows);
    }
}

$counter = new MysqlCounter('hostname', 'username', 'password');
$counter->gather_stats(array('MysqlCounter', 'default_callback'));
   
echo "Databases {$counter->num_databases}\n";
echo "Tables    {$counter->num_tables}\n";
echo "Rows      {$counter->num_rows}\n";
]]> http://project-2501.net/index.php/2009/09/total-mysql-rows/feed/ 0 WoW Combat Log Splitter http://project-2501.net/index.php/2009/09/wow-combat-log-splitter/ http://project-2501.net/index.php/2009/09/wow-combat-log-splitter/#comments Fri, 04 Sep 2009 07:55:11 +0000 Matthew http://project-2501.net/?p=199 4GB the WorldOfLogs parser will no longer do real time logging. Edit: Turns out that the WoW client itself stopped logging, even though the log file was a little over expected limit (4,334,806,196 bytes) Note: The code is just [...]]]> Something quick I whipped up last night, after noticing that after my log file was > 4GB the WorldOfLogs parser will no longer do real time logging.

Edit: Turns out that the WoW client itself stopped logging, even though the log file was a little over expected limit (4,334,806,196 bytes)

Note: The code is just a one off script; things are hard coded, and it’s pretty slow (100MB a minute)

#!/usr/bin/env python

import re
import datetime

GAP_SIZE_IN_SECONDS = 60 * 60

class CombatLog:
    def __init__(self, filename):
        self.filename = filename

    def process(self):
        last_timestamp = None
        line_count = 0
        split_log = None
       
        for line in open(self.filename):
            line_count += 1
            timestamp = self.parse_timestamp(line)
           
            if timestamp == None:
                print "Unparsable data on line %d" % (line_count,)
                print repr(line)
                print
                continue
           
            # To handle the first line
            if last_timestamp == None:
                last_timestamp = timestamp
                split_log = Appender(timestamp)
                print "Starting new file", split_log.filename
           
            difference = timestamp - last_timestamp
           
            if difference.seconds > GAP_SIZE_IN_SECONDS:
                # Close the old log file, and start a new one
                split_log.close()
                split_log = Appender(timestamp)
               
                print "Starting new file", split_log.filename

           
            split_log.append(line)
            last_timestamp = timestamp
           
            if line_count % 100000 == 0:
                print "Processed %d lines" % (line_count,)
           


    def parse_timestamp(self, line):
        # m/d hh:mm:ss.msec
        # 6/6 21:04:29.435
        regex = r"^(\d+)/(\d+) (\d+):(\d+):(\d+).(\d+) "
       
        matches = re.search(regex, line)
       
        if matches == None:
            return None
       
        timestamp = datetime.datetime(2009,
                                      int(matches.group(1)),
                                      int(matches.group(2)),
                                      int(matches.group(3)),
                                      int(matches.group(4)),
                                      int(matches.group(5)),
                                      int(matches.group(6))*1000);

        return timestamp


class Appender:
    def __init__(self, timestamp):
        self.filename = "WoWCombatLog_" + timestamp.strftime("%Y%m%d_%H%M%S") + ".txt"
        self.handle = open(self.filename, 'a')
   
    def append(self, line):
        self.handle.write(line)
       
    def close(self):
        self.handle.close()


splitter = CombatLog("../WoWCombatLog.20090904.txt")
splitter.process()

print "Done"
]]> http://project-2501.net/index.php/2009/09/wow-combat-log-splitter/feed/ 2