Practicing with Spark's join and scala on Array[String] -


i new both spark , scala, , i'm trying practice join command in spark.

i have 2 csv files:

ads.csv 

5de3ae82-d56a-4f70-8738-7e787172c018,adprovider1 f1b6c6f4-8221-443d-812e-de857b77b2f4,adprovider2 aca88cd0-fe50-40eb-8bda-81965b377827,adprovider1 940c138a-88d3-4248-911a-7dbe6a074d9f,adprovider3 983bb5e5-6d5b-4489-85b3-00e1d62f6a3a,adprovider3 00832901-21a6-4888-b06b-1f43b9d1acac,adprovider1 9a1786e1-ab21-43e3-b4b2-4193f572acbc,adprovider1 50a78218-d65a-4574-90de-0c46affbe7f3,adprovider5 d9bb837f-c85d-45d4-95f2-97164c62aa42,adprovider4 611cf585-a8cf-43e9-9914-c9d1dc30dab5,adprovider1

impression.csv is:

5de3ae82-d56a-4f70-8738-7e787172c018,publisher1 f1b6c6f4-8221-443d-812e-de857b77b2f4,publisher2 aca88cd0-fe50-40eb-8bda-81965b377827,publisher1 940c138a-88d3-4248-911a-7dbe6a074d9f,publisher3 983bb5e5-6d5b-4489-85b3-00e1d62f6a3a,publisher3 00832901-21a6-4888-b06b-1f43b9d1acac,publisher1 9a1786e1-ab21-43e3-b4b2-4193f572acbc,publisher1 611cf585-a8cf-43e9-9914-c9d1dc30dab5,publisher1

i want join them first id key , 2 values.

so read them in this:

val ads = sc.textfile("ads.csv") ads: org.apache.spark.rdd.rdd[string] = mappartitionsrdd[1] @ textfile  @ <console>:21 val impressions = sc.textfile("impressions.csv") impressions: org.apache.spark.rdd.rdd[string] = mappartitionsrdd[3] @ textfile @ <console>:21

ok, have make key,value pairs: val adpairs = ads.map(line => line.split(",")) val impressionpairs = impressions.map(line => line.split(","))

res11: org.apache.spark.rdd.rdd[array[string]] = mappartitionsrdd[6] @ map @ <console>:23 res13: org.apache.spark.rdd.rdd[array[string]] = mappartitionsrdd[7] @ map @ <console>:23

but can't join them:

val result = impressionpairs.join(adpairs) <console>:29: error: value join not member of       org.apache.spark.rdd.rdd[array[string]] val result = impressionpairs.join(adpairs)

do need convert pairs format?

you there, need transform array[string] key-value pairs, this:

val adpairs = ads.map(line => {   val substrings = line.split(",")   (substrings(0), substrings(1)) })

(and same impressionpairs)

that give rdds of type rdd[(string, string)] can joined :)


Comments

Post a Comment

Popular posts from this blog

authentication - Mongodb revoke acccess to connect test database -

i have ec2 instance mongodb 3.2.0 installed. have user admin access. can connect user , perform operation remotely. 1 can connect database test remote if knows ip mongo 11.11.11.11. although not able make operations, want restrict access user credentials can connect server. first need create users , grant roles users, example: use admin db.createuser( { user: "myuseradmin", pwd: "abc123", roles: [ { role: "useradminanydatabase", db: "admin" } ] } ) then need start server --auth parameter mongod --auth more information here
Read more

r - Update two sets of radiobuttons reactively - shiny -

i have read ( how make choices in radiobuttons reactive in shiny? ) shows me how update radiobutton s in reactive way. however, when try , update 2 sets of buttons same data, 1 set renders. example: server: # create example data wafer <- rep(c(1:3), each=3) length <- c(1,1,2,1,1,1,3,5,1) width <- c(3,1,6,1,1,1,1,1,6) dd <- data.frame(wafer, length, width) shinyserver(function(input, output, session){ # create reactive dataframe store data values <- reactivevalues() values$df <- data.frame() # lengths , widths of wafer user input <- eventreactive(input$do, { subset(dd, wafer %in% input$wafer, select = length:width) }) # update reactive data frame lengths , widths have been selected user input observe({ if(!is.null(a())) { values$df <- rbind(isolate(values$df), a()) } }) output$wl <- renderdatatable({ a() }) # update radio buttons unique length , widths stored in values$df # ever "observe" put first in co...
Read more

ios - Realm over CoreData should I use NSFetchedResultController or a Dictionary? -

i'm working on app using core data , nsfetchedresultcontroller many reasons switch use realm. saw there "realm version" of nsfetchedresultcontroller on github wouldn't compatible current code using core data. the view controller displaying list of people same school, address book. list sublist of people studied in same city. so thinking make 1 unique request database retrieve list of people , filter locally list within 1 dictionary per school [string: anyobject] string section name within tableview , anyobject array of people. first solution, make nsfetchedresultcontroller each school predicate filtering location. delete actions etc.. handled delegates -> not compatible realm create dictionaries , update them each actions... -> works realm it's annoying code. any better solution? edit: i need clarify request: i'd write class inherit uitableviewcontroller. this class has list of people sorted in alphabetical order the tableview has ...
Read more