Monday, July 1, 2013

Small script for one of the plarium game

few notes about google games.
Goggle informed that access to the all games be closed after June, 30. I played "Pirates: Tides of fortune" and decided to limit my game time using small script. Detail is below.
Some history
First internet games were updated network games only. 15 years ago internet connection requires a lot of money and nobody had a problem with lagging of the other players. Usually games uses personal communication protocol. Warbirds, Aces High and Eve online use this way.
Next part use a browser and trivial HTTP protocol. One of the popular game in this area is travian . Travian player uses browser and every browser (PC, cell phone, tablet) can be used for playing. Of course, popularity of the game is related to graphics. Trivial HTTP does not have good power in this and other technology is used for this side. One of them is Flash . Unfortunately flash requires a lot of CPU : "Pirates" was near to froze at my old celeron with 512MB. I hope next generation of browser's games will user WebGL.
Investigation
Modern HTTP games uses AJAX for sending data from client's application to the server. This one way decreases load at server side and increases client responses. I started to run ngrep session (ngrep is included in the most of linux distro) and capture some data from client to server and back:
POST /PiratesGp/Segment01/segment.ashx HTTP/1.1.
Host: 209.190.96.226.
Connection: keep-alive.
Content-Length: 158.
sign-code: d92d1d781e82623efc9125a18994876c.
Origin: https://rulesofwargame.com.
signin-userid: gpxxxxxxxxxxxxxxxxxxxxxxxxx
server-method: AutoRefresh.
User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.110 Safari/537.36.
client-ver: 405.
signin-authseed: fxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
signin-authkey: fxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.
locale-name: en-US.
content-type: text/html.
Accept: */*.
Accept-Encoding: gzip,deflate,sdch.
Accept-Language: en-US,en;q=0.8.
.
{"q":[1111,11111,11111,1111],"r":44007,"n":{"n":1},"y":null,"t":1372655066490,"u":1372653774360,"g":1061130,"chat":{"i":9353,"s":63507239220890,"r":["m.1","a.21"]}}


Look like games uses Google OAuth but I'm not sure. Anyway only next variables is important for developing bot: sign-code, signin-userid, signin-authseed, signin-authkey. Last three header fields are related to authorization. First variable ( sign-code) is interested. This one is used for security reason and is equal to md5sum from body and secret string. Body is JSON converted to string. Final python function is calculated as:
def generateRequestSignature(str1, str2):
    ''' Generate request signature  from '''
    return  hashlib.md5(("The Matrix has you..."+ str1+ str2).encode('ascii')).hexdigest()

After this trivial and long work should be done related to the client command. Each command should be investigated and added to the script. In the answer above "t" is a Unix time , "u" time from last login, "g" command id, "q" island position on the global map.
After this we can create a simple class for this game:
class PiratesClient:
    ''' Main class for generating daemon'''
    def __init__(self, profile): 
        ''' Constructor of the client. Set values of the headers and defaults paramaters fields.  
            Accept authSeed, authKey, userId, commandid, and user delta'''
        self.headers = { 
            "User-Agent": "Mozilla/5.0 (X11; Linux i686; rv:10.0.5) Gecko/20100101 Firefox/10.0.5 Iceweasel/10.0.5",
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", 
            "Accept-Language": "uk,ru;q=0.8,en-us;q=0.5,en;q=0.3",
            "Accept-Encoding": "gzip",
            "DNT": "1",
            "Connection": "keep-alive",
            "Content-Type": "text/html",
            "locale-name": "en-US"
        }
        #####################
        # read config file 
        #####################
        # system configuration
        self.config  = configparser.RawConfigParser()
        self.config.read('pirates.ini','UTF-8')

        # user configuration (dynamic create if not exists)
        self.config_user  = configparser.RawConfigParser()
        if os.path.exists('pirates_'+str(profile.my_uid)+'.ini'):
            self.config_user.read('pirates_'+str(profile.my_uid)+'.ini','UTF-8')
        else:
            with open('pirates_'+str(profile.my_uid)+'.ini', 'w') as configfile:
                self.config_user.add_section('client')
                self.config_user.set('client', 'commandid', "12345")            
                self.config_user.write(configfile)
                configfile.close()
                self.config_user.read('pirates_'+str(profile.my_uid)+'.ini','UTF-8')
         

        self.profile = profile
        self.commandId = int(self.config_user['client']['commandid'])
        self.version = int(self.config['game']['version'])
       
        self.headers["client-ver"] = self.version
        self.headers["signin-authSeed"] = profile.authSeed
        self.headers["signin-authKey"] = profile.authKey
        self.headers["signin-userId"] = profile.userId

        self.userdelta = 45678
        self.userId = profile.userId
        self.authKey = profile.authKey
        self.piratesurl = SERVER_IP
        self.userQ = profile.userQ
        self.userSegment = profile.userSegment
        self.params = {"q": self.userQ, "r":self.commandId, "v": self.version}
        self.connection = http.client.HTTPConnection(self.piratesurl)
        self.cachedUserRefresh = None

and creating gear at the heaven :
            
    def StartConversionJob(self):
        """  function for convert lumber and gold to gear (1 h for now)"""
        requesttime = round(time.time()*1000)
        new_g_value = round(requesttime-self.userdelta)
        self.commandId=self.commandId+1

        data = [{ # user ID list
            "t":requesttime,
            "u":new_g_value,
            "q":self.params["q"],
            "g":1089306,
            "r":self.commandId,
            "o":{"f":1, "i":3},
        }]

I think that creating other function is not so hard task. command
 ngrep -W byline -d eth0 host
helps to retrieve information. I have not made any changes at the client or server sides of this application. Simple result
Most of the function is already done. It is not a hard task but requires a lot of time. If you would like to create a similar game by yourself you can try. Server for this kind of game is priced under 20 $/month. (Original server uses Windows Vista Home edition, look like virtual host). Other is your time and initiative. You need some painter and musics skills fro creating frontend and some skill fro backend. Also I would like to say that game programming it is not gold mine , but you success in your hands. If anybody interested I can add few articles about creating this kind of games by using modern framework

I've add  few articles about investigating flash  games . First part is http://skhohlov.blogspot.com/2017/04/plarium-again.html

Wednesday, January 16, 2013

Python lists : performance or resource usage

Lists are native part of the Python language and this part makes programming easy and speedy. But every Moon has a dark side and I would like to add some light to it. Problem of the list is heavy resource's usage. Everyone should keep in mind this during coding. Simple example from python tutorial:
myfile = open("myfile.txt")
myfile.readlines()
Python opens file and creates a list from each line in it. Simple script below provides some information about executing speed and memory usage:
#!/usr/bin/python
import datetime
import resource

currenttime = datetime.datetime.now()
print "="*20
print "Creating a file "
print "="*20
myfile = open("textfile.txt", "w")
simplerange = xrange(10000000)
try:
    for i in simplerange:
        myfile.write(unicode(datetime.datetime.now()))
        myfile.write('\n')
finally:
    myfile.close()
timespend = datetime.datetime.now()- currenttime
print timespend
print "="*20


print "="*20
print "Open file using readlines"
print "="*20
myfile = open("textfile.txt", "r")
linesinlistfile = open("linesinthelist.txt", "w")
currenttime = datetime.datetime.now()
linesinlist = myfile.readlines()
for currentline in linesinlist:
    linesinlistfile.write(currentline)

myfile.close()
linesinlistfile.close()
myf = open("linesinthelist.txt", "r")

timespend = datetime.datetime.now()- currenttime
print timespend
print "="*20
print "openfile using readline"
print "="*20
myfile = open("textfile.txt", "r")
readonelinefile = open("readonelinefile.txt", "w")

while 1: 
    currentline = myfile.readline()
    if not currentline: break
    readonelinefile.write(currentline)
        
myfile.close()
readonelinefile.close()
timespend = datetime.datetime.now()- currenttime
print timespend
print "="*20
print "Resource usage"
print "="*20
print resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
This script creates a simple text file with time string in it, reads it using readline() and readlines() functions.  Last part returns memory usage in kilobytes.  For correct data I've commented part of codes related to readline or readlines.
Result is below:

readline() readlines()
executing time 0:01:10.799743 0:00:04.562637
memory usage 3620 526464
If list is used then performance is good but memory usage is really bad. Is it possible to have a good performance and good speed ? Lets try. There are two problems are present :
  1. Big list requires a lot of memory
  2. Solution without list can not be cached and be quick
But we can use small list approximately 1000 elements: read 1000 strings, make list, work with it, take another portion of the data.
while 1:    
    linesinlist = myfile.readlines(1000)
    if not linesinlist:
        break
    for currentline in linesinlist:
        linesinlistfile.write(currentline)
 
and result is below:
====================
Open file using readlines
====================
0:00:04.383583
====================
Resource usage
====================
3636
It is not hard to make good application, you should feel like it only !

Friday, January 11, 2013

Small python script for monitoring MySQL performance

I have few services which use MySQL as database server. I would like to have information about load in PNG  image  or  in Cacti app.
MySQL   has  performance information at 'SHOW STATUS' command.

Values  which  are monitored : 
 threads_running, threads_connected, thread_cached, slow_queries
 Of course,  it is really easy to add more variables.

Connection to MySQL is accomplished by MySQLdb  module. Typical example of usage is below :
import MySQLdb
mydb = MySQLdb.connect(host = 'hostname', 
                        user = 'username',
                        password = 'secret',
                        database = 'mysatabase'
)
mycursor = mydb.cursor()
mycursor.execute('SQL command')
sqlresult = cur.fetchall()

Storing data in rrd file is aviable via rrdtools package. This one is present in debian and Centos OS. example of creating file is below:
import rrdtool
rrdtool.create("myfile.rrd" ,
"DS:value1:datatype:heartbeat:lowerlimit:upperlimit ", 
"RRA:functionname:percentage:dataset:storedvalues")
This one function is more interested : "DS:value1:datatype:heartbeat:lowerlimit:upperlimit "means : value1 -- value which is stored in RRD heartbeat - howmuch time we wait before setting data to unknown lowerlimit:upperlimit - this is limits for a data "RRA:functionname:percentage:dataset:storedvalues" Functionname can be : AVERAGE the average of the data points is stored. MIN the smallest of the data points is stored. MAX the largest of the data points is stored. LAST the last data points is used. Percentage - how much unknown values can be but calculation be performed. dataset : how many values is used for calculation storedvalues : how many data are stored. Example: we are storing 5 min data and one day should be stored. In this case we need : 60/5 * 24 = 288 if we need information for week and with one hour interval then : 24 records for day (every hour) * 7 day = 168 Unite all together (Config parser is trivial and I skip it ):
#!/usr/bin/python
import MySQLdb
import sys
import rrdtool
from ConfigParser import SafeConfigParser

def main(conf_file="./mysqlmonitor.conf"):

    mydb, rrdfilename  = databaseconnect(conf_file)    

    cur = mydb.cursor()
    
    command = cur.execute('SHOW STATUS')
    res = cur.fetchall()

    for record in res:

        if record[0] == "Threads_running":
            threads_running = record[1]
            print "Threads_running:", threads_running
        if record[0] == "Threads_connected":
            threads_connected = record[1]
            print "Threads_connected:", threads_connected
        if record[0] == "Threads_cached":
            threads_cached  = record[1]
            print "Threads_cached:", threads_cached
        if record[0] == "Slow_queries":
            slow_queries = record[1]
            print slow_queries
            
                
    mydb.close()
    try:
        with open(rrdfilename) as rrdfile :
            rrdupdate(rrdfilename, threads_running, threads_connected, threads_cached, slow_queries)
    except IOError as e:
        print 'RRD file is not present creating'
        rrdcreate(rrdfile)

def rrdcreate(rrdfilename):
    """ function for creating RRD file"""
    ret = rrdtool.create(rrdfilename, "--step", "300", "--start", "0",
    "DS:threads_running:GAUGE:600:U:U",
    "DS:threads_connected:GAUGE:600:U:U",
    "DS:threads_cached:GAUGE:600:U:U",
    "DS:slow_queries:GAUGE:600:U:U",
    "RRA:AVERAGE:0.5:1:600",
    "RRA:AVERAGE:0.5:6:700",
    "RRA:AVERAGE:0.5:24:775",
    "RRA:MAX:0.5:1:600",
    "RRA:MAX:0.5:6:700",
    "RRA:MAX:0.5:444:797")


def rrdupdate(rrdfilename, threads_running, threads_connected, threads_cached, slow_queries):
    """ updating rrd data withnew information"""
    ret = rrdtool.update(rrdfilename, "N:%s:%s:%s:%s" %(threads_running, threads_connected, threads_cached, slow_queries))
    print "Updating"


def databaseconnect(conf_file=""):
    """ Log conf file and connect to database"""
    if not conf_file:
        sys.exit(-1)
    
    config = SafeConfigParser()
    config.read(conf_file)
    mydb = MySQLdb.connect(
        host = config.get('database','host'),
        user = config.get('database','user'),
        passwd = config.get('database','password'),
        db = 'INFORMATION_SCHEMA'
    )
    workingpath = config.get('files', 'rrd')
    return mydb, workingpath

if __name__ == "__main__":
    main()

Not bad for 3 hours of work!