python - Performance issue with Pandas merge function -


i merging multiple csv files using pandas' merge function takes more 3 hours process. note have more 1k files merge , columns can different though "security" column common in each file , values of unique. tips increase performance? doing wrong or inefficient. thank much! 

def consolidate_data_files (file_names, thread_count): output_file_names = []  joined_frame = none file_count = 0 file in file_names:     data_frame = pandas.read_csv(str(file), quoting=csv.quote_none, dtype=str)     if file_count == 0:         joined_frame = data_frame     else:         joined_frame = data_frame.merge(joined_frame, how='outer')     file_count += 1 total_row_count = len(joined_frame.index)  row_per_file = math.ceil(total_row_count/thread_count) merged_file_count = int(math.ceil(total_row_count/row_per_file);  in range(merged_file_count):     file = "merged_file_"+str(i)+".csv"     output_file_names.append(file)     row_start = int(i * row_per_file)     row_end = int(row_start + row_per_file)     joined_frame[row_start:row_end].to_csv(path_or_buf=file, index=false, quoting=csv.quote_none) del joined_frame return output_file_names


Comments

Post a Comment

Popular posts from this blog

php - Wordpress website dashboard page or post editor content is not showing but front end data is showing properly -

Image
recently have uploaded wordpress site onto server. have uploaded wordpress site in sub-folder under root folder ../root(domain root folder)/wordpress (where host wordpress folder). now problem content data in page or post in dashboard of wordpress admin panel not showing - see image. but data showing in front view - see image. if not browser cache issue , not issue happened tinymce editor during upgrade; try add config file: define('concatenate_scripts', false);
Read more

How to get the ip address of VM and use it to configure SSH connection dynamically in Ansible -

i trying ip address of vm , trying use establish ssh vm. following script: --- - hosts: localhost become: yes connection: local gather_facts: false serial: 1 vars_files: - createvmvars.yml tasks: - name: gathering vm info. vsphere_guest: vcenter_hostname: "{{vcenter_hostname}}" username: "{{vcenter_username}}" password: "{{vcenter_password}}" guest: "{{guest_name}}" vmware_guest_facts: yes register: var - debug: msg="{{var.ansible_facts.hw_eth0.ipaddresses[0]}}" - name: establishing ssh connection. script: /home/shasha/devops/scripts/ssh_configure.sh "{{var.ansible_facts.hw_eth0.ipaddresses[0]}}" following content of ssh_configure.sh : #!/bin/sh ssh-keygen -t rsa ssh-copy-id $1 but when running playbook, getting following error while executing third task: error! 'var' undefined but debug module(second task) printin
Read more

javascript - Get parameter of GET request -

i make request: createxhrrequest( "get", fileurl, function( err, response ) { if( err ) { alert( "error get!" ); } alert(response); }); and full response: { "status" : "ok", "message" : "jvberi0xljqkjdduxdgkmy how can jvberi0xlj ? i try response.message , response["message"]. nothing works. just simple access this: var json = json.parse(response); json.message
Read more