Agenda / Session
Session - DataOps Summit 2022The Oracle of Lake: Predicting Big Data Processing Time at Goldman Sachs
How machine learning is used to estimate ingestion and export times for petabytes of data running through Goldman Sachs' Data Lake platform.
Predicting processing time across a petabyte-scale platform
Data Lake is the firm's big data platform: a proprietary data store plus services and infrastructure components used to house and process petabytes of data every day.
The platform scales to run hundreds of thousands of ingestions and more than a million exports per day, and the team is constantly growing. Fast processing times matter to the business, so the work of forecasting them became a data problem in its own right.
The session covers the use of machine learning to estimate ingestion and export times for petabytes of data, giving the Data Lake platform a practical way to anticipate how long a given job will take before it runs.
What the session covers
- The Data Lake platform: proprietary data store, services, and infrastructure components.
- Machine learning models used to predict ingestion and export processing times.
- Operating at hundreds of thousands of ingestions and more than a million exports per day.
Presented at the summit
Jaimita Bansal
Vice President / Lead Data Scientist, Goldman Sachs
About the speaker
Jaimita Bansal is Vice President and Lead Data Scientist at Goldman Sachs, where she works on the Data Lake platform that houses and processes petabytes of data every day.
More speakers from Goldman Sachs, Snowflake, Humana, MIT and other organizations are listed on the speakers page.
At a glance
- SessionThe Oracle of Lake: Predicting Big Data Processing Time at Goldman Sachs
- SpeakerJaimita Bansal, Vice President / Lead Data Scientist, Goldman Sachs
- SummitDataOps Summit 2022, August 30 - 31, 2022, SF Bay Area
- FormatKeynote and breakout sessions, with an expo hall and hackathon
See the full schedule on the agenda page, and other sessions in the events section.