Databricks vs. AWS Redshift: Data Platform Comparison

The quantity of structured and unstructured data that enterprises must deal with today is such that most require the best in databases and data warehouses. A large data warehouse or data lake is needed, where both structured and unstructured data can be gathered, so analysts are free to investigate any data they wish at once—whether small slices or vast amounts.
Accordingly, cloud-based data platforms such as Databricks and Amazon Web Services (AWS) Redshift have emerged to meet these needs. Both are well-respected and highly rated by users on Gartner Peer Reviews. But which is best for your business?
Redshift and Databricks provide the volume, speed, and quality demanded by business intelligence (BI) applications. But there are as many similarities as there are differences. Therefore, selection often boils down to platform preference and suitability for your organization’s data strategy.
Jump to:

Databricks vs. Redshift: Key Features
Databricks vs. Redshift: Support and Ease of Use
Databricks vs. Redshift: Security
Databricks vs. Redshift: Integration
Databricks vs. Redshift: Pricing
Choosing Between Databricks and Redshift for Database Management

Databricks vs. Redshift: Key Features
Redshift
Redshift positions itself as a petabyte-scale data warehouse service that can be used by BI tools for analysis. Some of its best features include:

Redshift scales up and down easily.
Amazon offers independent clusters for load balancing to enhance performance.
Redshift offers good query performance—courtesy of high-bandwidth connections, proximity to users due to the many Amazon data centers around the world, and tailored communication protocols.
Amazon provides many services that enable easy access to reliable backups for Redshift datasets.

Databricks
Databricks is in the cloud but is based on Apache Spark. Its management layer is built around Apache Spark’s distributed computing framework to make management of infrastructure easier. Some of Databricks’ defining features include:

It uses a batch in-stream data processing engine for distribution across multiple nodes.
As a data lake, Databricks’ emphasis is more on use cases such as streaming, machine learning, and data science-based analytics.
It can be used on raw unprocessed data in large volumes.
Databricks is delivered as software as a service (SaaS) and can run on AWS, Azure, and Google Cloud.
There is a data plane as well as a control plane for back-end services that delivers instant compute.
Databricks’ query engine is said to offer high performance via a caching layer.
Databricks provides storage by running on top of AWS S3, Azure Blob Storage, and Google Cloud Storage.

Which Is Best for Its Features?
When it comes to comparing features, there is no clear winner between Redshift and Databricks. The best platform will depend on the organization and its database management needs.
For those wanting a top-class data warehouse for analytics, Redshift wins. But for those needing more robust ELT (extract, load, transform), data science, and machine learning features, Databricks is the winner.
For more information, also see: Best Data Analytics Tools 
Databricks vs. Redshift: Support and Ease of Use
Redshift
Amazon Redshift is said to be user-friendly and demands little administration for everyday use:

Set up, integration, and query running are easy for those already storing data on Amazon S3.
Redshift supports multiple data output formats, including JSON.
Those with a background in SQL will find it easy to harness PostgreSQL to work with data.
That said, some users noted that Redshift can sometimes be complex to set up and use at times and ties up more IT time on maintenance due to lack of automation. A lack of flexibility in areas such as resizing can lead to extra expense and long hours of maintenance. Amazon also requires some copying and other plumbing. It lacks support for some semi-structured data types.

Databricks
Databricks offers a variety of support options that can be used for technical and developer use cases:

Databricks can run Python, Spark Scholar, SQL, NC SQL, and other platforms.
It comes with its own user interface as well as ways to connect to endpoints such as Java database connectivity (JDBC) connectors.

Some users, though, report that it can appear complex and not user-friendly, as it is aimed at a technical market and needs more manual input for resizing clusters or configuration updates. There may be a steep learning curve for some.
Which Is Best for Support and Ease of Use?
This one is close although Redshift is the narrow winner.
For more information, also see: Top Data Visualization Tools 
Databricks vs. Redshift: Security
Redshift
Redshift does a good job on security and compliance. These features are enforced comprehensively for all users.
Additionally, tools are available for access management, cluster encryption, security groups for clusters, data encryption in transit and at rest, SSL connection security, and sign-in credential security. These tools enable security teams to monitor network access and traffic for any irregularities that might indicate a breach.
Access rights are granular and can be localized. Thus, Redshift makes it easy to restrict inbound or outbound access to clusters. The network can also be isolated within a virtual private cloud (VPC) and linked to the IT infrastructure via a virtual private network (VPN).
Databricks
Databricks provides role-based access control (RBAC) and automatic encryption and plenty of other security features.
Which Is Best for Security?
Both platforms do a good job of security, so there is no clear winner in this category.
Databricks vs. Redshift: Integration
Redshift
Obviously, those already committed to the AWS platforms will find integration seamless on Redshift with services like Athena, DMS, DynamoDB, and CloudWatch. The level of integration within AWS is excellent.
Databricks
In comparison, Databricks requires some third-party tools and application programming interface (API) configurations to integrate governance and data lineage features. Databricks, however, supports any format of data including unstructured data. But, it lacks the vendor partnership depth and breadth that Amazon can muster.
Which Is Best for Integration?
Integration: Redshift wins.
To learn more, also see: Top Business Intelligence Software 
Databricks vs. Redshift: Pricing
Redshift
Redshift provides a dedicated amount of daily concurrency scaling. But you get charged by the second if it is exceeded. Customers can be charged an hourly rate by type and cluster nodes or by amount of byte scanning. That said, Redshift’s long-term contracts come with big discounts.
Roughly speaking, Redshift costs about 25 cents per hour. But, the rate of usage will vary tremendously depending on the workload. Some users say Redshift is less expensive for on-demand pricing and that large datasets cost more.
Databricks
Databricks takes a different approach to packaging its services. Compute pricing for Databricks is tiered and charged per unit of processing, with its lowest paid tier starting at $99 per month. However, there is a free version for those who want to test it out before upgrading to a paid plan.
Databricks may work out cheaper for some users, depending on the way the storage is used and the frequency of use. For example, consultant fees for those needing help are said to be expensive.
Which Is Best Based on Pricing?
This is a close one, as it varies from use case to use case, but Amazon Redshift gets the nod.
The differences between them make it difficult to do a full apples-to-apples comparison. Users are advised to assess the resources they expect to need to support their forecast data volume, amount of processing, and analysis requirements before making a purchasing decision.
To learn more, also see: Data Analytics Trends 
Choosing Between Databricks and Redshift for Data Workloads
Databricks and Redshift are both excellent data warehouses and data lakes for analysis purposes. Each has its pros and cons. It all comes down to usage patterns, data volumes, workloads, and data strategies.
Big AWS users would be best on Redshift due to better integration with the entire Amazon ecosystem. However, Redshift is said to not function well with live app databases, there is no separation of storage and compute which can add to costs, and there is a maximum number of nodes that can be added to a cluster. It is up to the user to determine via good research which of these two fine platforms will suit their data patterns best.
Overall, Databricks is well-suited to streaming, machine learning, artificial intelligence, and data science workloads—courtesy of its Spark engine, which enables use of multiple languages. It isn’t really a data warehouse at all. Its data platform is wider in scope with better capabilities than Redshift for ELT, data science, and machine learning. Users store data in managed object storage of their choice and doesn’t get involved in its pricing. It focuses on the data lake and data processing. But it is squarely aimed at data scientists and highly capable analysts.
In summary, Databricks wins for a technical audience, and Amazon wins for a less technically gifted user base. Databricks provides pretty much of the data management functionality offered by AWS Redshift. But, it isn’t as easy to use, has a steep learning curve, and requires plenty of maintenance. But it can address a wider set of data workloads and languages. And those familiar with Apache Spark will tend to gravitate towards Databricks.
AWS Redshift is best for users on the AWS platform that just want to deploy a good data warehouse rapidly without bogging down in configurations, data science minutia, or manual setup. It isn’t nearly as high-end as Databricks, which is aimed more at complex data engineering, ETL (extract, transform, and load), data science, and streaming workloads. But Redshift also integrates with various data loading and ETL tools and BI reporting, data mining, and analytics tools. The fact that Databricks can run Python, Spark Scholar, SQL, NC SQL, and more will certainly make it attractive to developers in those camps.
For more information, also see: Top Data Mining Tools 
The post Databricks vs. AWS Redshift: Data Platform Comparison appeared first on eWEEK.