You know how to build ETLs but can you rely on them? What happens when you want to run data pipelines on a schedule with no manual kickoff? How do you troubleshoot ETLs?
Zack McCarty, the Director of Financial Planning and Analysis from Qwick and Parker Goldman, the Business Analytics Manager from Cannonball, joined WayScript's CEO and Cofounder, Jesse Orshan to answer precisely these questions at the Data Pipelines 201 Discussion and Workshop last week.
Panel
During the panel, Zack and Parker shared their ETL battle stories and how they overcame common data pipeline challenges such as rate limits. You can watch the panel here:
Rethinking ETL Design
Afterwards, Jesse shared an ETL philosophy for redesigning ETL scripting for internal tools at scale.
Grab the entire presentation here:
Data Pipelines 201 Workshop
Finally, the group participated in a hands on workshop and learned how to build two internal tools. Watch the full recording here:
The first tool is an automation that uses a Selenium web scraper to scrape competitor sites, write the data to an Airtable base, and send a Slack message notifying the channel when records in the base are updated. You can grab the two templates used for that tool here:
The second tool is a Flask server that displays data on an endpoint and updates automatically on a custom cron schedule. Grab the base template here:
Still curious about implementing scalable data pipelines? Meet with us 1:1 to discuss your team’s specific use case.
Stay tuned for the next virtual session! If you haven’t tried out WayScript yourself yet, you can create a free account here.
