etl - Complex join with google dataflow -


i'm newbie, trying understand how might re-write batch etl process google dataflow. i've read of docs, run few examples.

i'm proposing new etl process driven business events (i.e. source pcollection). these trigger etl process particular business entity. etl process extract datasets source systems , pass results (pcollections) onto next processing stage. processing stages involve various types of joins (including cartesian , non-key joins, e.g. date-banded).

so couple of questions here:

(1) approach i'm proposing valid & efficient? if not better, havent seen presentations on real-world complex etl processes using google dataflow, simple scenarios.

are there "higher-level" etl products better fit? i've been keeping eye on spark , flink while.

our current etl moderately complex, though there 30 core tables (classic edw dimensions , facts), , ~1000 transformation steps. source data complex (roughly 150 oracle tables).

(2) complex non-key joins, how these handled?

i'm attracted google dataflow because of being api first , foremost, , parallel processing capabilities seem fit (we being asked move batch overnight incremental processing).

a worked example of dataflow use case push adoption forward!

thanks, mike s

it sounds dataflow fit. allow write pipeline takes pcollection of business events , performs etl. pipeline either batch (executed periodically) or streaming (executed whenever input data arrives).

the various joins part relatively expressible in dataflow. cartesian product, can @ using side inputs make contents of pcollection available input processing of each element in pcollection.

you can @ using groupbykey or cogroupbykey implement joins. these flatten multiple inputs, , allow accessing values same key in 1 place. can use combine.perkey compute associative , commutative combinations of elements associated key (eg., sum, min, max, average, etc.).

date-banded joins sound fit windowing allows write pipeline consumes windows of data (eg., hourly windows, daily windows, 7 day windows slide every day, etc.).

edit: mention groupbykey , cogroupbykey.


Comments

Post a Comment

Popular posts from this blog

php - Wordpress website dashboard page or post editor content is not showing but front end data is showing properly -

Image
recently have uploaded wordpress site onto server. have uploaded wordpress site in sub-folder under root folder ../root(domain root folder)/wordpress (where host wordpress folder). now problem content data in page or post in dashboard of wordpress admin panel not showing - see image. but data showing in front view - see image. if not browser cache issue , not issue happened tinymce editor during upgrade; try add config file: define('concatenate_scripts', false);
Read more

How to get the ip address of VM and use it to configure SSH connection dynamically in Ansible -

i trying ip address of vm , trying use establish ssh vm. following script: --- - hosts: localhost become: yes connection: local gather_facts: false serial: 1 vars_files: - createvmvars.yml tasks: - name: gathering vm info. vsphere_guest: vcenter_hostname: "{{vcenter_hostname}}" username: "{{vcenter_username}}" password: "{{vcenter_password}}" guest: "{{guest_name}}" vmware_guest_facts: yes register: var - debug: msg="{{var.ansible_facts.hw_eth0.ipaddresses[0]}}" - name: establishing ssh connection. script: /home/shasha/devops/scripts/ssh_configure.sh "{{var.ansible_facts.hw_eth0.ipaddresses[0]}}" following content of ssh_configure.sh : #!/bin/sh ssh-keygen -t rsa ssh-copy-id $1 but when running playbook, getting following error while executing third task: error! 'var' undefined but debug module(second task) printin
Read more

javascript - Get parameter of GET request -

i make request: createxhrrequest( "get", fileurl, function( err, response ) { if( err ) { alert( "error get!" ); } alert(response); }); and full response: { "status" : "ok", "message" : "jvberi0xljqkjdduxdgkmy how can jvberi0xlj ? i try response.message , response["message"]. nothing works. just simple access this: var json = json.parse(response); json.message
Read more