I hate firewalls. Hate them. If you are using pf and wish to block a specific ip address using a simple table and a block quick rule it turns out just adding an ip to the table isn’t enough… pf wont actually block a new ip if it already has state information about it, instead it continues to let all traffic through. The solution is to remove all the state rules for the ip too:

pfctl -t blacklist -T add 123.123.123.123
pfctl -k 123.123.123.123