python - My multi-threading code with django connection pool doesn't have any improvment -


i struggling multi-threading connection pool on django.

i know python threading has gil issue thought python threading enough improve performance if of work db i/o.

first tried implement small code prove thought.

simply explaining, code uses threadpool.apply_async() db connection pool set conn_max_age in settings.py.

with code, repeat controlling number of threads worker thread.

from multiprocessing            import pool threadpooltestwithdb_io    import models django.db                  import transaction import django import datetime import logging import g2stype   def addegm(pre, id_):     """     @summary: function inserts bundle of records tied foreign key      """     try:         transaction.atomic():              egmid = pre + "_" + str(id_)             egm = models.g2segm(egmid=egmid, egmlocation="localhost")             egm.save()              device = models.device(egm=egm,                           deviceid=1,                           deviceclass=g2stype.t_deviceclass.g2s_eventhandler,                           deviceactive=true)             device.save()              models.eventhandlerprofile(device=device, queuebehavior="a").save()             models.eventhandlerstatus(device=device).save()              i2 in range(1, 200):                 models.eventreportdata(device=device,                                        deviceclass=g2stype.t_deviceclass.g2s_communications,                                        deviceid=1,                                        eventcode="test",                                        eventtext="",                                        eventid=i2,                                        transactionid=0                                        ).save()              print "done %d" % id_       except exception e:         logging.root.exception(e)   if __name__ == "__main__":      django.setup()     logging.basicconfig()      print "start test"      tpool = pool.threadpool(processes=1)    #set number of processes      s = datetime.datetime.now()     in range(100):                    #set number of record bundles         tpool.apply_async(func=addegm, args=("a", i))      print "wait worker processes"     tpool.close()                                tpool.join()      e = datetime.datetime.now()     print "end test"      print "time measurement : %s" % (e-s,)      models.g2segm.objects.all().delete()    #remove records inserted while test -------------------------- # settings.py   databases = {              'default': {                          'engine': 'django.db.backends.oracle',                          'name': 'orcl',                          'user': 'test',                          'password': '1123',                          'host': '192.168.0.90',                          'port': '1521',                          'conn_max_age': 100,                          'options': {'threaded': true}                          }              } 

however, result came out don't have big difference between 1 thread worker , multi-thread works.

for example, takes 30.6 sec 10 threads , takes 30.4 sec 1 thread.

what did go wrong?

either have problems on database level. can prove execution query:

select /* +rule */     s1.username || '@' || s1.machine     || ' ( sid=' || s1.sid || ' ' || s1.program || ' )  blocking ' || s2.username || '@' || s2.machine || ' ( sid=' || s2.sid || ' ' || s2.program || ' ) ' blocking_status     v$lock l1, v$session s1, v$lock l2, v$session s2     s1.sid=l1.sid , s2.sid=l2.sid     , l1.block=1 , l2.request > 0     , l1.id1 = l2.id1     , l2.id2 = l2.id2 ; 

or there threads being blocked in python. (possibly on db driver level). attach gdb python process , execute thread apply bt.

and see.


Comments

Popular posts from this blog

php - Wordpress website dashboard page or post editor content is not showing but front end data is showing properly -

How to get the ip address of VM and use it to configure SSH connection dynamically in Ansible -

javascript - Get parameter of GET request -