A simple statistical test shows that average temperatures are very unlikely to increase due to “bad luck”. — Introduction We are currently experiencing an increase of the average temperature from one year to the next. But is this really statistically significant? The average temperature is varying a lot over time, so it might well be the case that the current situation is just “bad luck” and that we will…

How to speed up building your Big Data ETL pipelines and getting them into production — We have been hearing the slogan “Data is the new Gold” since a couple of years for now and many companies are heavily investing to follow down this route. Initially most companies believed that is was enough to hire a bunch of expensive data scientists to become a leader in…

Create your own Insights on Global Warming using publicly available Data. — You are now reading the third and last part of my mini series on analyzing publicly available data to research the climate change. Still, the idea is not to become a real expert in meteorology but to apply common sense with appropriate tooling to derive some insights, such that everyone…

Create your own Insights on Global Warming using publicly available Data. — This is the second part of a small series dedicated to perform some analytics on publicly available weather data in order to find significant indicators for the climate change, specifically for the global warming. I am by no means an expert for meteorology nor for climate models, but I have…

Create your own Insights on Global Warming using publicly available Data — The climate change currently is a hot topic, with many experts claiming a significant increase of the average temperature over the whole world. …

Many companies follow the hype of big data without understanding the implications of the technology. — I call myself a “Big Data Expert”. I have tamed many animals in the ever growing Hadoop zoo like HBase, Hive, Oozie, Spark, Kafka, etc… I helped companies to build and structure their Data Lake using appropriate subsets of these technologies. I like to wrangle with data from multiple sources…

Why neither Spark nor Pandas is better than the other. Or: Always chose the right tool for the right job. — Originally I wanted to write a single article for a fair comparison of Pandas and Spark, but it continued to grow until I decided to split this up. This is the second part of the small series. Spark vs Pandas, part 1 — Pandas Spark vs Pandas, part 2 —…

Why programming languages matter — In this third installment of the series “Pandas vs Spark” we will have a closer look at the programming languages and the implications of choosing one. Originally I wanted to write a single article for a fair comparison of Pandas and Spark, but it continued to grow until I decided…

Pushing the limits by scaling with Spark — Originally I wanted to write a single article for a fair comparison of Pandas and Spark, but it continued to grow until I decided to split this up. This is the second part of the small series. Spark vs Pandas, part 1 — Pandas Spark vs Pandas, part 2 —…