How to manually deploy 3rd party utility jar for Apache Spark cluster? -


i have apache spark cluster (multi-nodes) , manually deploy utility jars each spark node. should put these jars to? example: spark-streaming-twitter_2.10-1.6.0.jar

i know can use maven build fat jar including these jars, deploy these utilities manually. in way, programmers not have deploy these utilities jars.

any suggestion?

1, copy 3rd party jars reserved hdfs directory;
example hdfs://xxx-ns/user/xxx/3rd-jars/

2, in spark-submit, specify these jars using hdfs path;
hdfs: - executors pull down files , jars hdfs directory

--jars hdfs://xxx-ns/user/xxx/3rd-jars/xxx.jar

3, spark-submit not repleatly upload these jars

client: source , destination file systems same. not copying hdfs://xxx-ns/user/xxx/3rd-jars/xxx.jar

Comments

Post a Comment

Popular posts from this blog

php - Wordpress website dashboard page or post editor content is not showing but front end data is showing properly -

Image
recently have uploaded wordpress site onto server. have uploaded wordpress site in sub-folder under root folder ../root(domain root folder)/wordpress (where host wordpress folder). now problem content data in page or post in dashboard of wordpress admin panel not showing - see image. but data showing in front view - see image. if not browser cache issue , not issue happened tinymce editor during upgrade; try add config file: define('concatenate_scripts', false);
Read more

javascript - Get parameter of GET request -

i make request: createxhrrequest( "get", fileurl, function( err, response ) { if( err ) { alert( "error get!" ); } alert(response); }); and full response: { "status" : "ok", "message" : "jvberi0xljqkjdduxdgkmy how can jvberi0xlj ? i try response.message , response["message"]. nothing works. just simple access this: var json = json.parse(response); json.message
Read more

javascript - Twitter Bootstrap - how to add some more margin between tooltip popup and element -

Image
i have not tryed since can't understand space between tooltip , element made/added bootstrap, tooltip near element triggers tooltip event. i make tooltip opening little bit away element, add more margin tooltip , eleemnt mean. i understand how in css if possible , if possible make both right,left,top,bottom tooltips hope question clear. this how tooltips looks on app: and , how should looks after this default css tooltip on top : .tooltip.top { padding: 5px 0; margin-top: -3px; } you can override in own css : .tooltip.top { margin-top: -10px; } note code work work tooltip on top, you'll nedd adapt css 3 other orientations : .tooltip.top { margin-top: -10px; } .tooltip.right { margin-left: 10px; } .tooltip.bottom { margin-top: 10px; } .tooltip.left { margin-left: -10px; }
Read more