Perlbal as a reverse proxy

I recently needed to set up a new reverse proxy as an alternative to pound, and for no particularly good reason chose Perlbal.

The documentation is fairly good, but I didn’t find many (good) examples of working configurations. So I thought I’d include my very simple conf.

LOAD vhosts

# Management service via telnet
CREATE SERVICE mgmt
        SET role   = management
        SET listen = 127.0.0.1:16000
ENABLE mgmt

# Web server
CREATE POOL web
        POOL web ADD 10.10.42.41:80

# Trac server
CREATE POOL trac
        POOL trac ADD 10.10.42.42:80

CREATE SERVICE web_proxy
        SET role = reverse_proxy
        SET pool = web
ENABLE web_proxy

CREATE SERVICE trac_proxy
        SET role = reverse_proxy
        SET pool = trac
ENABLE trac_proxy

# Internally we use 'trac.internal.com' but externally it would
# be 'trac.external.com'.  So rather than creating a second
# virtual host on our trac webserver, we re-write the header
HEADER trac_proxy REMOVE Host
HEADER trac_proxy INSERT Host:trac.internal.com

# Listen on our external IP
CREATE SERVICE selector
        SET listen  = 100.110.120.130:80
        SET role    = selector
        SET plugins = vhosts

        VHOST external.com.au       = web_proxy
        VHOST www.external.com.au   = web_proxy
        VHOST trac.external.com.au  = trac_proxy
ENABLE selector

Posted by Matthew on June 9th, 2011

Filed under random | No Comments »

Firefox, proxies and DNS

Firefox by default does not use your proxy when making DNS requests. This can lead to a bit of confusion if your internal DNS servers are different from your public servers, thankfully there’s a simple fix:

  1. Navigate to: about:config
  2. Locate network.proxy.socks_remote_dns and set the value to true

Posted by Matthew on June 9th, 2011

Filed under random | No Comments »

Clevo P150HM Notes

Maybe this will help someone else with this laptop:

  • If the USB3.0 ports randomly stop working you just need to update the drivers. You can grab them from Station Drivers
  • Fn+1 Toggles the fan speed between 100% and normal

Posted by Matthew on May 30th, 2011

Filed under random | No Comments »

Servers and Stuff

I’ve migrated most of my sites / services away from Slicehost this week. I guess the trigger was a combination of the Rackspace migration news, and the desire for lower latency to the server. I’ve been with Slicehost for around 3 years with an almost perfect track record (at least when I don’t let a process use up all the memory and the OOM killer).

I was originally looking at Crucial Paradigm due to lots of positive feedback and good prices. However their prices are really only good for new customers as the double RAM offer is not extended to upgrades/download of plans, which could cause some problems if you ever want to resize your VPS. They also specifically disallow game servers in thier AUP (which is a problem as I’m currently running a Minecraft server).

I tested the new iiNet VPS which is located in iiNet’s WA data centre. While the pricing is good, I wasn’t unhappy with the performance of the Virtuozzo based system (I was unable to run a Minecraft server with a single user without the load > 4.0), and the latency from Melbourne wasn’t exactly great.

I’ve now switched to MammothVPS1 and so far it’s been a good experience. Their data centre is in Sydney and has a fairly low latency for Melbourne and Sydney users2 and their plan offerings are very flexible and are competitively priced for an Australian based server. They use Xen (like Slicehost) so there’s much less risk for over selling.

[1] Yes that’s a referral link.
[2] On a iiNet ADSL2+ connection in inner Melbourne the latency is just 23ms. A TPG ADSL2+ connection in outer Melbourne was much higher at ~60ms.

Posted by Matthew on May 25th, 2011

Filed under vps | 2 Comments »

Sources of Holy Power Generation

Note: The information present in this post is a couple of months out of date. I’ve not had the time to re-update WoL data and create new graphs, but thought it might be interesting share the data/code anyway

When choosing a Holy Paladin spec for 10 mans I was never completely sure about benefits of Tower of Radiance or Blessed Life. ToR seems like a safe choice, though the frequency that you DL/FoL your beacon seemed low. Blessed Life on the other hand was obviously a PvP talent, but it would generate free HP from some raid damage.

I set out to scrape the top 200 Paladins for each 10-N boss fights from World of Logs and record their source of HP generation. I created two graphs; one that looks at all 200 parses in general, and a second that only counts when the ability is present. e.g. Only 10 or so Paladins used CS on Cho’gall, and they generated very little HP overall, but if you just look at lose 10 parses, the average is 8. This is used as a crude way to see when certain talents are taken.

Overall

Only when abilities are present

I’ve included the python code used to generate the data for these graphs below. My general disclaimer applies (i.e. be afraid)

grab-details.py

import urllib2
import re

pages = {
    'magmaw-10-n': ['http://worldoflogs.com/rankings/players/Blackwing_Descent/Magmaw/10N/Holy_Paladin/',
                    'http://worldoflogs.com/rankings/players/Blackwing_Descent/Magmaw/10N/Holy_Paladin/?page=2',
                    'http://worldoflogs.com/rankings/players/Blackwing_Descent/Magmaw/10N/Holy_Paladin/?page=3',
                    'http://worldoflogs.com/rankings/players/Blackwing_Descent/Magmaw/10N/Holy_Paladin/?page=4',
                    'http://worldoflogs.com/rankings/players/Blackwing_Descent/Magmaw/10N/Holy_Paladin/?page=5'],
       
    'omnitron-10-n': ['http://worldoflogs.com/rankings/players/Blackwing_Descent/Omnitron_Defense_System/10N/Holy_Paladin/',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Omnitron_Defense_System/10N/Holy_Paladin/?page=2',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Omnitron_Defense_System/10N/Holy_Paladin/?page=3',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Omnitron_Defense_System/10N/Holy_Paladin/?page=4',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Omnitron_Defense_System/10N/Holy_Paladin/?page=5'],
   
    'chimaeron-10-n': ['http://worldoflogs.com/rankings/players/Blackwing_Descent/Chimaeron/10N/Holy_Paladin/',
                       'http://worldoflogs.com/rankings/players/Blackwing_Descent/Chimaeron/10N/Holy_Paladin/?page=2',
                       'http://worldoflogs.com/rankings/players/Blackwing_Descent/Chimaeron/10N/Holy_Paladin/?page=3',
                       'http://worldoflogs.com/rankings/players/Blackwing_Descent/Chimaeron/10N/Holy_Paladin/?page=4',
                       'http://worldoflogs.com/rankings/players/Blackwing_Descent/Chimaeron/10N/Holy_Paladin/?page=5'],
   
    'atramedes-10-n': ['http://worldoflogs.com/rankings/players/Blackwing_Descent/Atramedes/10N/Holy_Paladin/',
                       'http://worldoflogs.com/rankings/players/Blackwing_Descent/Atramedes/10N/Holy_Paladin/?page=2',
                       'http://worldoflogs.com/rankings/players/Blackwing_Descent/Atramedes/10N/Holy_Paladin/?page=3',
                       'http://worldoflogs.com/rankings/players/Blackwing_Descent/Atramedes/10N/Holy_Paladin/?page=4',
                       'http://worldoflogs.com/rankings/players/Blackwing_Descent/Atramedes/10N/Holy_Paladin/?page=5'],
       
    'maloriak-10-n': ['http://worldoflogs.com/rankings/players/Blackwing_Descent/Maloriak/10N/Holy_Paladin/',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Maloriak/10N/Holy_Paladin/?page=2',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Maloriak/10N/Holy_Paladin/?page=3',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Maloriak/10N/Holy_Paladin/?page=4',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Maloriak/10N/Holy_Paladin/?page=5'],
   
    'nefarian-10-n': ['http://worldoflogs.com/rankings/players/Blackwing_Descent/Nefarian/10N/Holy_Paladin/',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Nefarian/10N/Holy_Paladin/?page=2',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Nefarian/10N/Holy_Paladin/?page=3',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Nefarian/10N/Holy_Paladin/?page=4',
                      'http://worldoflogs.com/rankings/players/Blackwing_Descent/Nefarian/10N/Holy_Paladin/?page=5'],
       
    'halfus-10-n': ['http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Halfus_Wyrmbreaker/10N/Holy_Paladin/',
                    'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Halfus_Wyrmbreaker/10N/Holy_Paladin/?page=2',
                    'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Halfus_Wyrmbreaker/10N/Holy_Paladin/?page=3',
                    'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Halfus_Wyrmbreaker/10N/Holy_Paladin/?page=4',
                    'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Halfus_Wyrmbreaker/10N/Holy_Paladin/?page=5'],
   
    'valiona-10-n': ['http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Valiona_&_Theralion/10N/Holy_Paladin/',
                     'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Valiona_&_Theralion/10N/Holy_Paladin/?page=2',
                     'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Valiona_&_Theralion/10N/Holy_Paladin/?page=3',
                     'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Valiona_&_Theralion/10N/Holy_Paladin/?page=4',
                     'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Valiona_&_Theralion/10N/Holy_Paladin/?page=5'],
   
    'twilight-10-n': ['http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Twilight_Ascendant_Council/10N/Holy_Paladin/',
                      'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Twilight_Ascendant_Council/10N/Holy_Paladin/?page=2',
                      'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Twilight_Ascendant_Council/10N/Holy_Paladin/?page=3',
                      'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Twilight_Ascendant_Council/10N/Holy_Paladin/?page=4',
                      'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Twilight_Ascendant_Council/10N/Holy_Paladin/?page=5'],

    'chogall-10-n': ['http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Cho\'gall/10N/Holy_Paladin/',
                     'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Cho\'gall/10N/Holy_Paladin/?page=2',
                     'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Cho\'gall/10N/Holy_Paladin/?page=3',
                     'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Cho\'gall/10N/Holy_Paladin/?page=4',
                     'http://worldoflogs.com/rankings/players/Bastion_of_Twilight/Cho\'gall/10N/Holy_Paladin/?page=5'],
   
    'conclave-10-n': ['http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Conclave_of_Wind/10N/Holy_Paladin/',
                      'http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Conclave_of_Wind/10N/Holy_Paladin/?page=2',
                      'http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Conclave_of_Wind/10N/Holy_Paladin/?page=3',
                      'http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Conclave_of_Wind/10N/Holy_Paladin/?page=4',
                      'http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Conclave_of_Wind/10N/Holy_Paladin/?page=5'],
   
    'alakir-10-n': ['http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Al\'Akir/10N/Holy_Paladin/',
                    'http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Al\'Akir/10N/Holy_Paladin/?page=2',
                    'http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Al\'Akir/10N/Holy_Paladin/?page=3',
                    'http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Al\'Akir/10N/Holy_Paladin/?page=4',
                    'http://worldoflogs.com/rankings/players/Throne_of_the_4_Winds/Al\'Akir/10N/Holy_Paladin/?page=5'],
   
}

def extract_player_pages(url):
    regex   = "<td><a href='(/reports/[^']+)'>([^<]+)</a></td>"
    html    = urllib2.urlopen(url).read()
    matches = re.findall(regex, html)
   
    return matches

def extract_player_details(report_url, player_name):
    # First load the healing summary page
    url   = 'http://worldoflogs.com' + report_url
    html  = urllib2.urlopen(url).read()
    match = re.search("<a href='(/reports/[^']+)'>" + player_name + "</a>", html)
   
    # now load the players healing detail page
    url   = 'http://worldoflogs.com' + match.group(1)
    html  = urllib2.urlopen(url).read()
   
    return html


for boss in pages:
    print boss
   
    for url in pages[boss]:
        reports = extract_player_pages(url)
       
        for report_details in reports:
            report_url         = report_details[0]
            report_player_name = report_details[1]
           
            print 'Extracting', report_player_name
           
            html = extract_player_details(report_url, report_player_name)
            f    = open('data/' + boss + '/' + report_player_name, 'w+')
            f.write(html)
            f.close()
           
print 'Done'

generate-report.py

import glob
import re

# Spell IDs
HOLY_SHOCK         = 20473
ETERNAL_GLORY      = 88676
TOWER_OF_RADIANCE  = 88852
BLESSED_LIFE       = 89023
PURSUIT_OF_JUSTICE = 89024
CRUSADER_STRIKE    = 35395

bosses = [
    'alakir-10-n',
    'atramedes-10-n',
    'chimaeron-10-n',
    'chogall-10-n',
    'conclave-10-n',
    'halfus-10-n',
    'magmaw-10-n',
    'maloriak-10-n',
    'nefarian-10-n',
    'omnitron-10-n',
    'twilight-10-n',
    'valiona-10-n',
]


def extract_holy_power(html):
    regex   = r"            <td class='name'><a href='/reports/[^']+' rel='spell=(\d+)' class='spell'><span [^>]+>([^<]+)</span></a>\S+\n           <td>(\d+) holy power</td>"
    matches = re.findall(regex, html)
   
    results = {}
   
    for match in matches:
        (spell_id, spell_name, count) = match
        results[int(spell_id)] = int(count)
       
    return results


def generate_holy_power_summary(boss):
    summary = {}
   
    files = glob.glob('data/' + boss + '/*')
    for file in files:
        html = open(file, 'r').read()
        results = extract_holy_power(html)
       
        for spell_id in results:
            count = results[spell_id]
            if not summary.has_key(spell_id):
                summary[spell_id] = []
            summary[spell_id].append(count)
   
    return summary


report_on = [
    {'spell_name': 'HOLY_SHOCK', 'spell_id': HOLY_SHOCK},
    {'spell_name': 'ETERNAL_GLORY', 'spell_id': ETERNAL_GLORY},
    {'spell_name': 'TOWER_OF_RADIANCE', 'spell_id': TOWER_OF_RADIANCE},
    {'spell_name': 'BLESSED_LIFE', 'spell_id': BLESSED_LIFE},
    {'spell_name': 'PURSUIT_OF_JUSTICE', 'spell_id': PURSUIT_OF_JUSTICE},
    {'spell_name': 'CRUSADER_STRIKE', 'spell_id': CRUSADER_STRIKE},
]


for boss in bosses:
    print
    print boss
   
    summary = generate_holy_power_summary(boss)
   
    if not summary.has_key(HOLY_SHOCK):
        print 'Skipping, no data'
        continue
   
    total_count = len(summary[HOLY_SHOCK])
   
    print 'Total reports', total_count
   
    if total_count == 0:
        continue
   
    for report_details in report_on:
        if not summary.has_key(report_details['spell_id']):
            print report_details['spell_name'], 0
            continue
       
        spell_count   = len(summary[report_details['spell_id']])
        total_average = sum(summary[report_details['spell_id']]) / total_count
        spell_average = sum(summary[report_details['spell_id']]) / spell_count
       
        print report_details['spell_name'], total_average

Posted by Matthew on April 5th, 2011

Filed under games, programming, python | No Comments »

Text Twist Bot

Yesterday I watched someone play a bit of the browser based game Text Twist. Upon trying myself I found that I was awful, so I did what any programmer would do; cheated.

Results of a few hours hacking:

Features:

  • Scan for game area and crash and burn if it’s not detected
  • (Crudely) detect what letters are shown
  • Find all 3, 4, 5 and 6 letter combinations, and then throw them against a basic word list + simple anagram lookup table
  • Send key presses to the window to (try and) solve all possible words (and overwrite any work you were doing when the window loses focus)
  • Reliably gets all but one or two words for each puzzle, leaving you plenty of time to go crazy trying to finish it

Horrible source for the curious (it should at least help anyone wondering how to send key presses, or capture the current screen in python)

TextTwistBot.py

# Text Twist Bot
# http://games.yahoo.com/game/text-twist

import win32com.client as comclt
from time import sleep

from PIL import ImageGrab
from itertools import combinations

from Board import Board
from Pixels import Pixels
from Anagrams import Anagrams


def sorted_unique_character_groups(letters, length):
    c = list(combinations(''.join(letters), length))
    c = [''.join(e) for e in c]
    c = list(set(c))
    c.sort()
    return c


anagrams = Anagrams('ispell-enwl-3.1.20/english.all')
pixels   = Pixels(ImageGrab.grab())
board    = Board(pixels)
letters  = board.get_letters()

# letters = ['d', 'e', 'l', 'a', 'p', 'd']
possible_answers = []
lengths = [3,4,5,6]

for length in lengths:
    words = sorted_unique_character_groups(letters, length)
    for word in words:
        possible_answers.extend(anagrams.find(word))

possible_answers = list(set(possible_answers))

print possible_answers


wsh = comclt.Dispatch("WScript.Shell")
wsh.AppActivate("Windows Internet Explorer")

for word in possible_answers:
    for letter in word:
        wsh.SendKeys(letter)
        sleep(0.05)
    wsh.SendKeys("\n")

print 'Done'

Pixels.py

class Pixels:
    def __init__(self, img):
        self.img = img
        self.data = list(img.getdata())
        (self.width, self.height) = img.size
       
    def at(self, x, y):
        offset = (y * self.width) + x
        return self.data[offset]
       
    def grab_area(self, x1, y1, x2, y2):
        raw = []
        for y in range(y1, y2):
            for x in range(x1, x2):
                raw.append(self.at(x, y))
        return raw

Anagrams.py

class Anagrams:
    def __init__(self, word_list):
        self._load_word_list(word_list)
       
    def _load_word_list(self, filename):
        print 'Loading word list...',
        lines = open(filename).readlines()
        words = [line.strip().lower() for line in lines]
       
        self.lookup = {}
       
        for word in words:
            key = self._sort_letters(word)
            if key not in self.lookup:
                self.lookup[key] = []
            if word not in self.lookup[key]:
                self.lookup[key].append(word)
        print 'Done'
       
    def _sort_letters(self, word):
        letters = [l for l in word]
        letters.sort()
        return ''.join(letters)
       
    def find(self, word):
        key = self._sort_letters(word)
        if key not in self.lookup:
            return []
        results = self.lookup[key]
        #if word in results:
        #    results.remove(word)
        return results

Board.py (be afraid)

class Board:
    def __init__(self, pixels):
       
        # Color of the outside border, used to find the edges of the game
        self.border_green = (204, 255, 0)
       
        # Bounding boxes for each letter (including the circle and background)
        # Each letter is 45x45
        # I've chopped off 8pixels from each side to remove any background
        self.letter_coords = [
            ((161+8, 178+8), (161+45-8, 178+45-8)),
            ((215+8, 178+8), (215+45-8, 178+45-8)),
            ((269+8, 178+8), (269+45-8, 178+45-8)),
            ((323+8, 178+8), (323+45-8, 178+45-8)),
            ((377+8, 178+8), (377+45-8, 178+45-8)),
            ((431+8, 178+8), (431+45-8, 178+45-8)),
        ]
       
        # Pixel data for each letter
        self.letter_data = {
            # Snipped 300Kb of data
            # Yikes, didn't realize it was so big
            # If anyone actually cares for the letter data, you
            # can reconstruct it from error messages.
            #
            # Format is:
            #
            # 'a': [(r, g, b), (r, g, b)....],
            # 'b': [(r, g, b), (r, g, b)....],
        }
       
        self.pixels = pixels
        self._get_edges()
       
    def _get_edges(self):
        horz = self._find_pixels(range(0, self.pixels.width),      range(0, self.pixels.height, 100), self.border_green)
        vert = self._find_pixels(range(0, self.pixels.width, 100), range(0, self.pixels.height),      self.border_green)
       
        left   = min([x for (x,y) in horz])
        right  = max([x for (x,y) in horz])
        top    = min([y for (x,y) in vert])
        bottom = max([y for (x,y) in vert])
       
        self.left   = left
        self.right  = right
        self.top    = top
        self.bottom = bottom
       
    def _find_pixels(self, x_range, y_range, color):
        edges = []
        for y in y_range:
            for x in x_range:
                if self.pixels.at(x, y) == color:
                    edges.append((x, y))
        return edges
   
    def get_letters(self):
        letters = []
        for coords in self.letter_coords:
            x1 = coords[0][0] + self.left
            x2 = coords[1][0] + self.left
           
            y1 = coords[0][1] + self.top
            y2 = coords[1][1] + self.top
       
            data = self.pixels.grab_area(x1, y1, x2, y2)
           
            # print 'Got data for pos', coords
            found = False
            for key in self.letter_data:
                if self.letter_data[key] == data:
                    letters.append(key)
                    found = True
                    break
           
            if found == False:
                print '\'?\':', data, ','
        return letters

Posted by Matthew on March 16th, 2011

Filed under games, programming, python | 2 Comments »