python - Efficiently processing a very large unicode string into csv -


usually i'm able find answers dilemmas pretty on site perhaps problem requires more specific touch;

i have ~50 million long unicode string download tektronix oscilloscope. getting assigned pain in a** memory (sys.getsizeof() reports ~100 mb)

the problem lies in need turn csv can grab 10,000 of 10 million comma sep values (this fixed)... 1) have tried split(",") method, using this, ram usage on python kernel spikes 300 mb....but process efficient (except when loop ~100 times in 1 routine...somewhere between iterations 40-50, kernel spits memory error.) 2) wrote own script after downloading absurdly long string, scans the number of commas until see 10,000 , stops, turning values between commas floats , populating np array. pretty efficient memory usage perspective (from before importing file after running script, memory usage changes 150mb.) slower, , results in kernel crash shortly after completion of 100x loops.

below code used process file, , if pm me, can send copy of string experimenting (however i'm sure may easier generate one)

code 1 (using split() method)

ppstrace = ppsinst.query('curv?') ppstrace = ppstrace.split(',') ppsvals = [] iii in range(len(ppstrace)): #does algebra values     ppstrace[iii] = ((float(ppstrace[iii]))-yoff)*ymult+yzero  maxes=np.empty(shape=(0,0)) iters=int(samples/1000) in range(1000): #looks max value in 10,000 sample increments, adds "maxes"     print     maxes = np.append(maxes,max(ppstrace[i*iters:(i+1)*iters])) pps = 100*np.std(maxes)/np.mean(maxes) print pps," % pps noise"

code 2 (self generated script);

ppstrace = ppsinst.query('curv?') walkerr=1 walkerl=0 length=len(ppstrace) maxes=np.empty(shape=(0,0)) iters=int(samples/1000) #samples 10 million, iters 10000  in range(1000):     sample=[] #initialize 10k sample list     commas=0 #commas 0     while commas<iters: #if number of commas found less 10,000, keep adding values sample         while ppstrace[walkerr]!=unicode(","):#indexes commas value extraction             walkerr+=1             if walkerr==length:                 break         sample.append((float(str(ppstrace[walkerl:walkerr]))-yoff)*ymult+yzero)#add value between commas sample list         walkerl=walkerr+1         walkerr+=1         commas+=1     maxes=np.append(maxes,max(sample)) pps = 100*np.std(maxes)/np.mean(maxes) print pps,"% pps noise"

also tried pandas dataframe stringio csv conversion. thing gets memory error trying read frame.

i thinking solution load sql table , pull csv in 10,000 sample chunks (which intended purpose of script). love not this!

thanks guys!

have tried class cstringio? it's file io, uses string buffer instead of specified file. frankly, expect you're suffering chronic speed problem. self-generated script should right approach. might speed-up if read block @ time, , parse while next block reading.

for parallel processing, use multiprocessing package. see official documentation or this tutorial details , examples.

briefly, create function embodies process want run in parallel. create process function target parameter. start process. when want merge thread main program, use join.


Comments

Post a Comment

Popular posts from this blog

authentication - Mongodb revoke acccess to connect test database -

i have ec2 instance mongodb 3.2.0 installed. have user admin access. can connect user , perform operation remotely. 1 can connect database test remote if knows ip mongo 11.11.11.11. although not able make operations, want restrict access user credentials can connect server. first need create users , grant roles users, example: use admin db.createuser( { user: "myuseradmin", pwd: "abc123", roles: [ { role: "useradminanydatabase", db: "admin" } ] } ) then need start server --auth parameter mongod --auth more information here
Read more

r - Update two sets of radiobuttons reactively - shiny -

i have read ( how make choices in radiobuttons reactive in shiny? ) shows me how update radiobutton s in reactive way. however, when try , update 2 sets of buttons same data, 1 set renders. example: server: # create example data wafer <- rep(c(1:3), each=3) length <- c(1,1,2,1,1,1,3,5,1) width <- c(3,1,6,1,1,1,1,1,6) dd <- data.frame(wafer, length, width) shinyserver(function(input, output, session){ # create reactive dataframe store data values <- reactivevalues() values$df <- data.frame() # lengths , widths of wafer user input <- eventreactive(input$do, { subset(dd, wafer %in% input$wafer, select = length:width) }) # update reactive data frame lengths , widths have been selected user input observe({ if(!is.null(a())) { values$df <- rbind(isolate(values$df), a()) } }) output$wl <- renderdatatable({ a() }) # update radio buttons unique length , widths stored in values$df # ever "observe" put first in co...
Read more

ios - Realm over CoreData should I use NSFetchedResultController or a Dictionary? -

i'm working on app using core data , nsfetchedresultcontroller many reasons switch use realm. saw there "realm version" of nsfetchedresultcontroller on github wouldn't compatible current code using core data. the view controller displaying list of people same school, address book. list sublist of people studied in same city. so thinking make 1 unique request database retrieve list of people , filter locally list within 1 dictionary per school [string: anyobject] string section name within tableview , anyobject array of people. first solution, make nsfetchedresultcontroller each school predicate filtering location. delete actions etc.. handled delegates -> not compatible realm create dictionaries , update them each actions... -> works realm it's annoying code. any better solution? edit: i need clarify request: i'd write class inherit uitableviewcontroller. this class has list of people sorted in alphabetical order the tableview has ...
Read more