Take a deep breath. There are myriad options when it comes to data science stack technologies, which at first glance can be overwhelming for business leaders looking to implement the stack that best fits their business needs.
Ready for the good news?
There really isn’t a best data science stack – but there is the most effective option for your goals.
First, let’s talk about what we mean by data science, considering how rapidly the field evolves. At its core, data science is simply applying models to solve business problems and inform business decisions, but where those models come from and how they are applied can inform which stack you employ.
Getting Started: Understanding the Role Data Science Plays
At the end of the day, what do you need your stack to provide? Specific focuses might include:
- Analysis
- Visualization
- Machine Learning
- Creating tools that enable data analysis (data engineering)
- Cloud availability, for data and tools
- Application delivery
- Automated processes
By understanding the end goal, you can better select the tool that will get you there. A typical workflow for a data scientist generally includes developing an idea, identifying the right data to use, locating that data, cleaning the data, and then finally, developing and testing the model. All these tasks leading up the model tend to be the most time consuming occupying the data scientists’ expertise elsewhere.
Enter the data science stack, an infrastructure that, when designed effectively can streamline the process and speed up the research cycle.
Data Science Software Stack
Given the wide variety of data stack technology on the market, it’s important to keep in mind the end goal: freeing data scientists to spend more time focused on the modeling, and ensuring they are working with the highest possible data quality.
Once you understand your end goal, the next step is understanding that every stack is unique. At the most basic level, stacks generally include some, if not all, of these elements:
- Database
- ETL (extraction, transformation, and loading)
- Warehouse/Storage and data modeling
- Visualizations (A business intelligence tool that shares insights)
- Artificial Intelligence/Machine Learning
You should also consider four key areas that will help you eliminate options that simply aren’t a fit for your business. These areas are:
- Does your business want on-premise or cloud-based services, or a combination of the two?
- Do you have the development professionals at your organization to create models and analytics functions, or do you need that to be provided?
- Are you already using a cloud service provider somewhere else in the business?
- Do your data analytics need to be conducted in real time?
Here are some common data science stack examples that might be helpful in understanding how to get started.
Languages and Databases
Open-source tools are less likely to offer the flexibility that custom solutions can provide but will require you to build your own frameworks and schedulers.
- Python. Used for scripting and automation languages, Python is often lauded for being easy to both learn and use. According to InfoWorld, Python helps developers focus on business problems rather than coding problems. Python is widely available, across major operating systems and platforms, and can interface with API-powered services as well. Python can be used for scripting, automation, or even commands for machine learning libraries and algorithms.
- MySQL. This is an open source database management system that leverages SQL to access, manage, and manipulate the database. MySQL is generally flexible, easy to use, and quick to process.
- PostgreSQL. This open source database management system supports advanced data types and optimizations generally found only in pricey commercial databases, such as SQL Server or Oracle. You may also see it referred to by its original name, Postgres. For building applications with solid data integrity, PostgreSQL’s widespread compatibility and zero-cost (yes, it’s free) can support complex queries, especially valuable for data warehousing and analysis applications that need quick read-write capabilities.
Data Engineering
- Airflow. This workflow management open source platform is especially equipped to execute complex dependencies across numerous systems, because of a plethora of plugins that can interact with most external systems. In practice, that might mean loading website analytics data into a data warehouse every hour, or aggregating sales updates from Salesforce directly dashboards and reports.
Data Warehousing
One important consideration for data warehousing is whether you’re looking for an on-premise or cloud based solution. Cloud-based software or platform as a service solutions are appealing because they are generally maintenance-free, and can free up resources for other focuses.
- Snowflake. This cloud-based solution is delivered as SaaS – meaning no need for hardware or software installation, configuration, and management. This flexibility enables even small organizations to move data into Snowflake, to maximize on the storage and computation available – or to focus on just one, if both storage and computation aren’t essential.
API Tools
- Embulk. This open-source data loader helps transfer data between database storage or cloud services. With an ability to predict file formats and execute big datasets, Embulk is equipped to handle unwieldy data. In practice, businesses often need to fetch and sync data sources in various formats (CSV, MySQL, JSON, etc.) from multiple locations to other databases, warehouses, or files.
Data Analysis
- Redash. This cloud-based platform enables small and midsized businesses to query data sources, specify role-based access, automate alerts, and create data visualizations for business intelligence, essentially making complex data accessible to anyone within an organization.
- Metabase. This platform can be either cloud-based or on-premise, and it helps businesses with KPI tracking, dashboarding, database management, record filtering, and query building. As a self-hosted software, it requires manual updates.
Some parting words
It’s not possible to learn every technology on the market today (really, it’s probably not even possible to count them). To refine your data science stack, you should:
- Understand what your objectives are
- Weigh the pros and cons of specific technologies
- Narrow your focus to a few essential tools
- Understand what common knowledge is required to use a tool in the stack
- Prioritize and minimize technology stack
Leveraging existing infrastructure can also help simplify business processes – if the organization is already using a data warehouse, there is no need to develop a separate infrastructure for data scientists. In fact, using a common infrastructure can eliminate the burden of sourcing, verifying, and cleaning the data and allow other facets of the organization to focus on the data warehouse maintenance, freeing data scientists to focus on their core value add: making models and algorithms.
Looking to learn more about data science stacks? Watch our On-Demand webinar entitled, “Data Science: Effective Use Cases & ROI.”
Sources:
- https://dev.to/minchulkim87/my-data-science-tech-stack-2020-1poa
- https://towardsdatascience.com/building-your-data-science-technology-stack-8c740e2db797
- https://towardsdatascience.com/building-the-ai-stack-ad23ac90405f
- https://www.123-reg.co.uk/support/servers/what-is-mysql-and-why-do-i-need-it/#:~:text=MySQL%20is%20a%20relational%20database,on%20SQL%20%E2%80%93%20Structured%20Query%20Language.&text=The%20most%20common%20use%20for,products%20for%20an%20online%20store.
- https://www.infoworld.com/article/3204016/what-is-python-powerful-intuitive-programming.html
- https://www.guru99.com/introduction-postgresql.html
- https://www.astronomer.io/guides/intro-to-airflow
- https://towardsdatascience.com/some-common-data-science-stacks-7525477c0dd0
- https://fivetran.com/blog/modern-data-science-stack
- https://hevodata.com/learn/building-a-data-science-tech-stack/
