python - My multi-threading code with django connection pool doesn't have any improvment -
i struggling multi-threading connection pool on django
.
i know python threading has gil
issue thought python threading enough improve performance if of work db i/o.
first tried implement small code prove thought.
simply explaining, code uses threadpool.apply_async()
db connection pool set conn_max_age
in settings.py
.
with code, repeat controlling number of threads worker thread.
from multiprocessing import pool threadpooltestwithdb_io import models django.db import transaction import django import datetime import logging import g2stype def addegm(pre, id_): """ @summary: function inserts bundle of records tied foreign key """ try: transaction.atomic(): egmid = pre + "_" + str(id_) egm = models.g2segm(egmid=egmid, egmlocation="localhost") egm.save() device = models.device(egm=egm, deviceid=1, deviceclass=g2stype.t_deviceclass.g2s_eventhandler, deviceactive=true) device.save() models.eventhandlerprofile(device=device, queuebehavior="a").save() models.eventhandlerstatus(device=device).save() i2 in range(1, 200): models.eventreportdata(device=device, deviceclass=g2stype.t_deviceclass.g2s_communications, deviceid=1, eventcode="test", eventtext="", eventid=i2, transactionid=0 ).save() print "done %d" % id_ except exception e: logging.root.exception(e) if __name__ == "__main__": django.setup() logging.basicconfig() print "start test" tpool = pool.threadpool(processes=1) #set number of processes s = datetime.datetime.now() in range(100): #set number of record bundles tpool.apply_async(func=addegm, args=("a", i)) print "wait worker processes" tpool.close() tpool.join() e = datetime.datetime.now() print "end test" print "time measurement : %s" % (e-s,) models.g2segm.objects.all().delete() #remove records inserted while test -------------------------- # settings.py databases = { 'default': { 'engine': 'django.db.backends.oracle', 'name': 'orcl', 'user': 'test', 'password': '1123', 'host': '192.168.0.90', 'port': '1521', 'conn_max_age': 100, 'options': {'threaded': true} } }
however, result came out don't have big difference between 1 thread worker , multi-thread works.
for example, takes 30.6 sec
10 threads , takes 30.4 sec
1 thread.
what did go wrong?
either have problems on database level. can prove execution query:
select /* +rule */ s1.username || '@' || s1.machine || ' ( sid=' || s1.sid || ' ' || s1.program || ' ) blocking ' || s2.username || '@' || s2.machine || ' ( sid=' || s2.sid || ' ' || s2.program || ' ) ' blocking_status v$lock l1, v$session s1, v$lock l2, v$session s2 s1.sid=l1.sid , s2.sid=l2.sid , l1.block=1 , l2.request > 0 , l1.id1 = l2.id1 , l2.id2 = l2.id2 ;
or there threads being blocked in python. (possibly on db driver level). attach gdb python process , execute thread apply bt
.
and see.
Comments
Post a Comment