Skip to main content

Python lists : performance or resource usage

Lists are native part of the Python language and this part makes programming easy and speedy. But every Moon has a dark side and I would like to add some light to it. Problem of the list is heavy resource's usage. Everyone should keep in mind this during coding. Simple example from python tutorial:
myfile = open("myfile.txt")
myfile.readlines()
Python opens file and creates a list from each line in it. Simple script below provides some information about executing speed and memory usage:
#!/usr/bin/python
import datetime
import resource

currenttime = datetime.datetime.now()
print "="*20
print "Creating a file "
print "="*20
myfile = open("textfile.txt", "w")
simplerange = xrange(10000000)
try:
    for i in simplerange:
        myfile.write(unicode(datetime.datetime.now()))
        myfile.write('\n')
finally:
    myfile.close()
timespend = datetime.datetime.now()- currenttime
print timespend
print "="*20


print "="*20
print "Open file using readlines"
print "="*20
myfile = open("textfile.txt", "r")
linesinlistfile = open("linesinthelist.txt", "w")
currenttime = datetime.datetime.now()
linesinlist = myfile.readlines()
for currentline in linesinlist:
    linesinlistfile.write(currentline)

myfile.close()
linesinlistfile.close()
myf = open("linesinthelist.txt", "r")

timespend = datetime.datetime.now()- currenttime
print timespend
print "="*20
print "openfile using readline"
print "="*20
myfile = open("textfile.txt", "r")
readonelinefile = open("readonelinefile.txt", "w")

while 1: 
    currentline = myfile.readline()
    if not currentline: break
    readonelinefile.write(currentline)
        
myfile.close()
readonelinefile.close()
timespend = datetime.datetime.now()- currenttime
print timespend
print "="*20
print "Resource usage"
print "="*20
print resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
This script creates a simple text file with time string in it, reads it using readline() and readlines() functions.  Last part returns memory usage in kilobytes.  For correct data I've commented part of codes related to readline or readlines.
Result is below:

readline() readlines()
executing time 0:01:10.799743 0:00:04.562637
memory usage 3620 526464
If list is used then performance is good but memory usage is really bad. Is it possible to have a good performance and good speed ? Lets try. There are two problems are present :
  1. Big list requires a lot of memory
  2. Solution without list can not be cached and be quick
But we can use small list approximately 1000 elements: read 1000 strings, make list, work with it, take another portion of the data.
while 1:    
    linesinlist = myfile.readlines(1000)
    if not linesinlist:
        break
    for currentline in linesinlist:
        linesinlistfile.write(currentline)
 
and result is below:
====================
Open file using readlines
====================
0:00:04.383583
====================
Resource usage
====================
3636
It is not hard to make good application, you should feel like it only !

Comments

Popular posts from this blog

Upgrade postgres 9.4 to 9.6 at Debian linux

I'm using virtual server based on Debian Linux for some my free projects.  One of them is based on Postgres with PostGIS extension.   This windy rainy day is a good time for executing some upgrade. Also I would like to add partman extension for partituoning.
Checking new version. Update package list :
aptitude update Get: 1 http://ftp.debian.org/debian jessie-updates InRelease [145 kB] Get: 2 http://security.debian.org stretch/updates InRelease [62.9 kB] Get: 3 http://security.debian.org jessie/updates InRelease [63.1 kB]
 Check for new version:


dpkg-query -l postgresql* Desired=Unknown/Install/Remove/Purge/Hold| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)||/ Name Version Ar…

Small script for one of the plarium game

few notes about google games.
Goggle informed that access to the all games be closed after June, 30. I played "Pirates: Tides of fortune" and decided to limit my game time using small script. Detail is below.
Some history
First internet games were updated network games only. 15 years ago internet connection requires a lot of money and nobody had a problem with lagging of the other players. Usually games uses personal communication protocol. Warbirds, Aces High and Eve online use this way.
Next part use a browser and trivial HTTP protocol. One of the popular game in this area is travian . Travian player uses browser and every browser (PC, cell phone, tablet) can be used for playing. Of course, popularity of the game is related to graphics. Trivial HTTP does not have good power in this and other technology is used for this side. One of them is Flash . Unfortunately flash requires a lot of CPU : "Pirates" was near to froze at my old celeron with 512M…

Small python script for monitoring MySQL performance

I have few services which use MySQL as database server. I would like to have information about load in PNG  image  or  in Cacti app.
MySQL   has  performance information at 'SHOW STATUS' command.

Values  which  are monitored : 
 threads_running, threads_connected, thread_cached, slow_queries
 Of course,  it is really easy to add more variables.

Connection to MySQL is accomplished by MySQLdb  module. Typical example of usage is below :
import MySQLdb mydb = MySQLdb.connect(host = 'hostname', user = 'username', password = 'secret', database = 'mysatabase' ) mycursor = mydb.cursor() mycursor.execute('SQL command') sqlresult = cur.fetchall()
Storing data in rrd file is aviable via rrdtools package. This one is present in debian and Centos OS. example of creating file is below: import rrdtool rrdtool.create("myfile.rrd" , "DS:value1:datatype:hea…