python - Performance issue with Pandas merge function -


i merging multiple csv files using pandas' merge function takes more 3 hours process. note have more 1k files merge , columns can different though "security" column common in each file , values of unique. tips increase performance? doing wrong or inefficient. thank much! 

def consolidate_data_files (file_names, thread_count): output_file_names = []  joined_frame = none file_count = 0 file in file_names:     data_frame = pandas.read_csv(str(file), quoting=csv.quote_none, dtype=str)     if file_count == 0:         joined_frame = data_frame     else:         joined_frame = data_frame.merge(joined_frame, how='outer')     file_count += 1 total_row_count = len(joined_frame.index)  row_per_file = math.ceil(total_row_count/thread_count) merged_file_count = int(math.ceil(total_row_count/row_per_file);  in range(merged_file_count):     file = "merged_file_"+str(i)+".csv"     output_file_names.append(file)     row_start = int(i * row_per_file)     row_end = int(row_start + row_per_file)     joined_frame[row_start:row_end].to_csv(path_or_buf=file, index=false, quoting=csv.quote_none) del joined_frame return output_file_names


Comments

Post a Comment

Popular posts from this blog

php - Wordpress website dashboard page or post editor content is not showing but front end data is showing properly -

Image
recently have uploaded wordpress site onto server. have uploaded wordpress site in sub-folder under root folder ../root(domain root folder)/wordpress (where host wordpress folder). now problem content data in page or post in dashboard of wordpress admin panel not showing - see image. but data showing in front view - see image. if not browser cache issue , not issue happened tinymce editor during upgrade; try add config file: define('concatenate_scripts', false);
Read more

javascript - Get parameter of GET request -

i make request: createxhrrequest( "get", fileurl, function( err, response ) { if( err ) { alert( "error get!" ); } alert(response); }); and full response: { "status" : "ok", "message" : "jvberi0xljqkjdduxdgkmy how can jvberi0xlj ? i try response.message , response["message"]. nothing works. just simple access this: var json = json.parse(response); json.message
Read more

javascript - Twitter Bootstrap - how to add some more margin between tooltip popup and element -

Image
i have not tryed since can't understand space between tooltip , element made/added bootstrap, tooltip near element triggers tooltip event. i make tooltip opening little bit away element, add more margin tooltip , eleemnt mean. i understand how in css if possible , if possible make both right,left,top,bottom tooltips hope question clear. this how tooltips looks on app: and , how should looks after this default css tooltip on top : .tooltip.top { padding: 5px 0; margin-top: -3px; } you can override in own css : .tooltip.top { margin-top: -10px; } note code work work tooltip on top, you'll nedd adapt css 3 other orientations : .tooltip.top { margin-top: -10px; } .tooltip.right { margin-left: 10px; } .tooltip.bottom { margin-top: 10px; } .tooltip.left { margin-left: -10px; }
Read more