what factors affect how many spark job concurrently -


we have set spark job server spark jobs submitted.but found out our 20 nodes(8 cores/128g memory per node) spark cluster can afford 10 spark jobs running concurrently.

can share detailed info factors affect how many spark jobs can run concurrently? how can tune conf can take full advantage of cluster?

question missing context, first - seems spark job server limits number of concurrent jobs (unlike spark itself, puts limit on number of tasks, not jobs):

from application.conf

 # number of jobs can run simultaneously per context  # if not set, defaults number of cores on machine jobserver running  max-jobs-per-context = 8

if that's not issue (you set limit higher, or using more 1 context), total number of cores in cluster (8*20 = 160) maximum number of concurrent tasks. if each of jobs creates 16 tasks, spark queue next incoming job waiting cpus available.

spark creates task per partition of input data, , number of partitions decided according partitioning of input on disk, or calling repartition or coalesce on rdd/dataframe manually change partitioning. other actions operate on more 1 rdd (e.g. union) may change number of partitions.


Comments

Post a Comment

Popular posts from this blog

php - Wordpress website dashboard page or post editor content is not showing but front end data is showing properly -

Image
recently have uploaded wordpress site onto server. have uploaded wordpress site in sub-folder under root folder ../root(domain root folder)/wordpress (where host wordpress folder). now problem content data in page or post in dashboard of wordpress admin panel not showing - see image. but data showing in front view - see image. if not browser cache issue , not issue happened tinymce editor during upgrade; try add config file: define('concatenate_scripts', false);
Read more

How to get the ip address of VM and use it to configure SSH connection dynamically in Ansible -

i trying ip address of vm , trying use establish ssh vm. following script: --- - hosts: localhost become: yes connection: local gather_facts: false serial: 1 vars_files: - createvmvars.yml tasks: - name: gathering vm info. vsphere_guest: vcenter_hostname: "{{vcenter_hostname}}" username: "{{vcenter_username}}" password: "{{vcenter_password}}" guest: "{{guest_name}}" vmware_guest_facts: yes register: var - debug: msg="{{var.ansible_facts.hw_eth0.ipaddresses[0]}}" - name: establishing ssh connection. script: /home/shasha/devops/scripts/ssh_configure.sh "{{var.ansible_facts.hw_eth0.ipaddresses[0]}}" following content of ssh_configure.sh : #!/bin/sh ssh-keygen -t rsa ssh-copy-id $1 but when running playbook, getting following error while executing third task: error! 'var' undefined but debug module(second task) printin
Read more

javascript - Get parameter of GET request -

i make request: createxhrrequest( "get", fileurl, function( err, response ) { if( err ) { alert( "error get!" ); } alert(response); }); and full response: { "status" : "ok", "message" : "jvberi0xljqkjdduxdgkmy how can jvberi0xlj ? i try response.message , response["message"]. nothing works. just simple access this: var json = json.parse(response); json.message
Read more