Connect with us

AI in Travel

Building Modern Data Lakehouses on Google Cloud with Apache Iceberg and Apache Spark

Published

2 weeks ago

on


Sponsored Content

 

 

 

The landscape of big data analytics is constantly evolving, with organizations seeking more flexible, scalable, and cost-effective ways to manage and analyze vast amounts of data. This pursuit has led to the rise of the data lakehouse paradigm, which combines the low-cost storage and flexibility of data lakes with the data management capabilities and transactional consistency of data warehouses. At the heart of this revolution are open table formats like Apache Iceberg and powerful processing engines like Apache Spark, all empowered by the robust infrastructure of Google Cloud.

 

The Rise of Apache Iceberg: A Game-Changer for Data Lakes

 

For years, data lakes, typically built on cloud object storage like Google Cloud Storage (GCS), offered unparalleled scalability and cost efficiency. However, they often lacked the crucial features found in traditional data warehouses, such as transactional consistency, schema evolution, and performance optimizations for analytical queries. This is where Apache Iceberg shines.

Apache Iceberg is an open table format designed to address these limitations. It sits on top of your data files (like Parquet, ORC, or Avro) in cloud storage, providing a layer of metadata that transforms a collection of files into a high-performance, SQL-like table. Here’s what makes Iceberg so powerful:

  • ACID Compliance: Iceberg brings Atomicity, Consistency, Isolation, and Durability (ACID) properties to your data lake. This means that data writes are transactional, ensuring data integrity even with concurrent operations. No more partial writes or inconsistent reads.
  • Schema Evolution: One of the biggest pain points in traditional data lakes is managing schema changes. Iceberg handles schema evolution seamlessly, allowing you to add, drop, rename, or reorder columns without rewriting the underlying data. This is critical for agile data development.
  • Hidden Partitioning: Iceberg intelligently manages partitioning, abstracting away the physical layout of your data. Users no longer need to know the partitioning scheme to write efficient queries, and you can evolve your partitioning strategy over time without data migrations.
  • Time Travel and Rollback: Iceberg maintains a complete history of table snapshots. This enables “time travel” queries, allowing you to query data as it existed at any point in the past. It also provides rollback capabilities, letting you revert a table to a previous good state, invaluable for debugging and data recovery.
  • Performance Optimizations: Iceberg’s rich metadata allows query engines to prune irrelevant data files and partitions efficiently, significantly accelerating query execution. It avoids costly file listing operations, directly jumping to the relevant data based on its metadata.

By providing these data warehouse-like features on top of a data lake, Apache Iceberg enables the creation of a true “data lakehouse,” offering the best of both worlds: the flexibility and cost-effectiveness of cloud storage combined with the reliability and performance of structured tables.

Google Cloud’s BigLake tables for Apache Iceberg in BigQuery offers a fully-managed table experience similar to standard BigQuery tables, but all of the data is stored in customer-owned storage buckets. Support features include:

  • Table mutations via GoogleSQL data manipulation language (DML)
  • Unified batch and high throughput streaming using the Storage Write API through BigLake connectors such as Spark
  • Iceberg V2 snapshot export and automatic refresh on each table mutation
  • Schema evolution to update column metadata
  • Automatic storage optimization
  • Time travel for historical data access
  • Column-level security and data masking

Here’s an example of how to create an empty BigLake Iceberg table using GoogleSQL:


SQL

CREATE TABLE PROJECT_ID.DATASET_ID.my_iceberg_table (
  name STRING,
  id INT64
)
WITH CONNECTION PROJECT_ID.REGION.CONNECTION_ID
OPTIONS (
file_format="PARQUET"
table_format="ICEBERG"
storage_uri = 'gs://BUCKET/PATH');

 

You can then import data into the data using LOAD INTO to import data from a file or INSERT INTO from another table.


SQL

# Load from file
LOAD DATA INTO PROJECT_ID.DATASET_ID.my_iceberg_table
FROM FILES (
uris=['gs://bucket/path/to/data'],
format="PARQUET");

# Load from table
INSERT INTO PROJECT_ID.DATASET_ID.my_iceberg_table
SELECT name, id
FROM PROJECT_ID.DATASET_ID.source_table

 

In addition to a fully-managed offering, Apache Iceberg is also supported as a read-external table in BigQuery. Use this to point to an existing path with data files.


SQL

CREATE OR REPLACE EXTERNAL TABLE PROJECT_ID.DATASET_ID.my_external_iceberg_table
WITH CONNECTION PROJECT_ID.REGION.CONNECTION_ID
OPTIONS (
  format="ICEBERG",
  uris =
    ['gs://BUCKET/PATH/TO/DATA'],
  require_partition_filter = FALSE);

 

 

Apache Spark: The Engine for Data Lakehouse Analytics

 

While Apache Iceberg provides the structure and management for your data lakehouse, Apache Spark is the processing engine that brings it to life. Spark is a powerful open-source, distributed processing system renowned for its speed, versatility, and ability to handle diverse big data workloads. Spark’s in-memory processing, robust ecosystem of tools including ML and SQL-based processing, and deep Iceberg support make it an excellent choice.

Apache Spark is deeply integrated into the Google Cloud ecosystem. Benefits of using Apache Spark on Google Cloud include:

  • Access to a true serverless Spark experience without cluster management using Google Cloud Serverless for Apache Spark.
  • Fully managed Spark experience with flexible cluster configuration and management via Dataproc.
  • Accelerate Spark jobs using the new Lightning Engine for Apache Spark preview feature.
  • Configure your runtime with GPUs and drivers preinstalled.
  • Run AI/ML jobs using a robust set of libraries available by default in Spark runtimes, including XGBoost, PyTorch and Transformers.
  • Write PySpark code directly inside BigQuery Studio via Colab Enterprise notebooks along with Gemini-powered PySpark code generation.
  • Easily connect to your data in BigQuery native tables, BigLake Iceberg tables, external tables and GCS
  • Integration with Vertex AI for end-to-end MLOps

 

Iceberg + Spark: Better Together

 

Together, Iceberg and Spark form a potent combination for building performant and reliable data lakehouses. Spark can leverage Iceberg’s metadata to optimize query plans, perform efficient data pruning, and ensure transactional consistency across your data lake.

Your Iceberg tables and BigQuery native tables are accessible via BigLake metastore. This exposes your tables to open source engines with BigQuery compatibility, including Spark. 


Python

from pyspark.sql import SparkSession

# Create a spark session
spark = SparkSession.builder \
.appName("BigLake Metastore Iceberg") \
.config("spark.sql.catalog.CATALOG_NAME", "org.apache.iceberg.spark.SparkCatalog") \
.config("spark.sql.catalog.CATALOG_NAME.catalog-impl", "org.apache.iceberg.gcp.bigquery.BigQueryMetastoreCatalog") \
.config("spark.sql.catalog.CATALOG_NAME.gcp_project", "PROJECT_ID") \
.config("spark.sql.catalog.CATALOG_NAME.gcp_location", "LOCATION") \
.config("spark.sql.catalog.CATALOG_NAME.warehouse", "WAREHOUSE_DIRECTORY") \
.getOrCreate()
spark.conf.set("viewsEnabled","true")

# Use the blms_catalog
spark.sql("USE `CATALOG_NAME`;")
spark.sql("USE NAMESPACE DATASET_NAME;")

# Configure spark for temp results
spark.sql("CREATE namespace if not exists MATERIALIZATION_NAMESPACE");
spark.conf.set("materializationDataset","MATERIALIZATION_NAMESPACE")

# List the tables in the dataset
df = spark.sql("SHOW TABLES;")
df.show();

# Query the tables
sql = """SELECT * FROM DATASET_NAME.TABLE_NAME"""
df = spark.read.format("bigquery").load(sql)
df.show()
sql = """SELECT * FROM DATASET_NAME.ICEBERG_TABLE_NAME"""
df = spark.read.format("bigquery").load(sql)
df.show()

sql = """SELECT * FROM DATASET_NAME.READONLY_ICEBERG_TABLE_NAME"""
df = spark.read.format("bigquery").load(sql)
df.show()

 

Extending the functionality of BigLake metastore is the Iceberg REST catalog (in preview) to access Iceberg data with any data processing engine. Here’s how to connect to it using Spark:


Python

import google.auth
from google.auth.transport.requests import Request
from google.oauth2 import service_account
import pyspark
from pyspark.context import SparkContext
from pyspark.sql import SparkSession

catalog = ""
spark = SparkSession.builder.appName("") \
    .config("spark.sql.defaultCatalog", catalog) \
    .config(f"spark.sql.catalog.{catalog}", "org.apache.iceberg.spark.SparkCatalog") \
    .config(f"spark.sql.catalog.{catalog}.type", "rest") \
    .config(f"spark.sql.catalog.{catalog}.uri",
"https://biglake.googleapis.com/iceberg/v1beta/restcatalog") \
    .config(f"spark.sql.catalog.{catalog}.warehouse", "gs://") \
    .config(f"spark.sql.catalog.{catalog}.token", "") \
    .config(f"spark.sql.catalog.{catalog}.oauth2-server-uri", "https://oauth2.googleapis.com/token") \                   .config(f"spark.sql.catalog.{catalog}.header.x-goog-user-project", "") \     .config("spark.sql.extensions","org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
.config(f"spark.sql.catalog.{catalog}.io-impl","org.apache.iceberg.hadoop.HadoopFileIO") \    .config(f"spark.sql.catalog.{catalog}.rest-metrics-reporting-enabled", "false") \
.getOrCreate()

 

 

Completing the lakehouse

 

Google Cloud provides a comprehensive suite of services that complement Apache Iceberg and Apache Spark, enabling you to build, manage, and scale your data lakehouse with ease while leveraging many of the open-source technologies you already use:

  • Dataplex Universal Catalog: Dataplex Universal Catalog provides a unified data fabric for managing, monitoring, and governing your data across data lakes, data warehouses, and data marts. It integrates with BigLake Metastore, ensuring that governance policies are consistently enforced across your Iceberg tables, and enabling capabilities like semantic search, data lineage, and data quality checks.
  • Google Cloud Managed Service for Apache Kafka: Run fully-managed Kafka clusters on Google Cloud, including Kafka Connect. Data streams can be read directly to BigQuery, including to managed Iceberg tables with low latency reads.
  • Cloud Composer: A fully managed workflow orchestration service built on Apache Airflow.
  • Vertex AI: Use Vertex AI to manage the full end-to-end ML Ops experience. You can also use Vertex AI Workbench for a managed JupyterLab experience to connect to your serverless Spark and Dataproc instances.

 

Conclusion

 

The combination of Apache Iceberg and Apache Spark on Google Cloud offers a compelling solution for building modern, high-performance data lakehouses. Iceberg provides the transactional consistency, schema evolution, and performance optimizations that were historically missing from data lakes, while Spark offers a versatile and scalable engine for processing these large datasets.

To learn more, check out our free webinar on July 8th at 11AM PST where we’ll dive deeper into using Apache Spark and supporting tools on Google Cloud.

Author: Brad Miro, Senior Developer Advocate – Google

 
 



Source link

Related Topics:
Continue Reading
Click to comment

You must be logged in to post a comment Login

Leave a Reply

AI in Travel

India-France Partnership to Build Drones for Defence and Global Exports

Published

1 hour ago

on

July 19, 2025

By


RRP Defence (RRP Group), through its entity Vimananu, has entered into a strategic partnership with the Franco-American firm CYGR to establish an advanced drone manufacturing facility in India. 

The project, based in Navi Mumbai, aims to support India’s ‘Make in India’ initiative by developing unmanned aerial vehicles (UAVs) for defence, surveillance, and industrial applications.

The collaboration will manufacture three categories of drones: hand-launched fixed-wing drones for field operations, compact nano drones for close-range use, and ISR drones tailored for intelligence, surveillance, and reconnaissance missions. 

Production is expected to start with hundreds of units annually, with an initial contract valued at over $20 million.

Rajendra Chodankar, chairman of RRP Defence, said, “This collaboration is a defining moment for India’s UAV ecosystem. By combining our local manufacturing strength and field understanding with CYGR’s world-class drone technologies, we’re building systems that meet India’s unique operational needs.”

The facility will also contribute to high-skill employment and export-focused manufacturing, further positioning India as a key player in the global UAV supply chain.

The initiative strengthens India’s self-reliance in aerospace and defence technology. Zaynah, the global advisor in Make in India global exports in defence for the collaboration, confirmed that an immediate Letter of Intent (LoI) is being released as part of the $20 million contract for global defence exports.

George El Aily, director of CYGR France, added, “India is a key strategic partner for us. Through this collaboration with RRP Defence Ltd, we are not only transferring technology but also co-developing future-ready solutions that support India’s defence and surveillance landscape.”

The project will focus on sectors including defence, homeland security, and industrial monitoring. The companies aim to deliver solutions customised for India’s operational environments while expanding global market access through technology localisation.

This joint venture marks a step forward in India’s ambition to become a global drone hub through international cooperation and indigenous expertise.



Source link

Continue Reading

AI in Travel

Figure AI Unveils In-House Battery for F03 Humanoid with 5-Hour Runtime

Published

1 hour ago

on

July 19, 2025

By


Figure AI, a California-based robotics company, introduced its third-generation battery for its latest F03 humanoid robot, marking a significant step in its in-house hardware development. 

The new battery offers 2.3 kWh of energy, supports five hours of peak performance runtime, and includes built-in safety systems to prevent thermal runaway. The battery is manufactured entirely in-house at Figure’s BotQ facility, which is now ramping up production capabilities.

Founder Brett Adcock said in a LinkedIn post that Figure “succeeds when we own the full stack,” with every core system, including actuators and batteries, engineered and built internally. 

The battery is the first in the humanoid robotics space to be in the process of securing both UN38.3 and UL2271 safety certifications.

The F03 battery is directly integrated into the robot’s torso, unlike the earlier external battery packs of the F01. It utilises structural components such as stamped steel and die-cast aluminium, allowing it to act as a load-bearing part of the robot and saving space and weight.

The company claims a 94% increase in energy density over its first-generation battery and a 78% cost reduction compared to the previous F.02 model. The battery’s structural and thermal design features active cooling, a flame arrestor vent, and a custom Battery Management System (BMS) that helps prevent fault conditions, such as overheating or short circuits.

The battery is manufactured using mass production techniques, including stamping, injection moulding, and die casting, which enables Figure to target production volumes of up to 12,000 humanoid units per year.

To support manufacturing scale-up at its BotQ facility, Figure has opened roles across several verticals, including software test engineering, electrical testing, manufacturing engineering, and equipment handling.

Recently, Adcock had also said that the company has tripled the team to 293 people to support manufacturing, supply chain, and fleet operations.

According to the company’s blog, Figure has collaborated with OSHA-accredited testing labs to establish safety standards for humanoid robots, as these standards did not previously exist. “We specified that the battery system of F.03 must not emit flames should a catastrophic failure of a single cell occur,” Figure said.





Source link

Continue Reading

AI in Travel

7 Python Web Development Frameworks for Data Scientists

Published

21 hours ago

on

July 18, 2025

By


7 Python Web Development Frameworks
Image by Author | Canva

 

Python is widely known for its popularity among engineers and data scientists, but it’s also a favorite choice for web developers. In fact, many developers prefer Python over JavaScript for building web applications because of its simple syntax, readability, and the vast ecosystem of powerful frameworks and tools available.

Whether you are a beginner or an experienced developer, Python offers frameworks that cater to every need, from lightweight micro-frameworks that require just a few lines of code, to robust full-stack solutions packed with built-in features. Some frameworks are designed for rapid prototyping, while others focus on security, scalability, or lightning-fast performance. 

In this article, we will review seven of the most popular Python web frameworks. You will discover which ones are best suited for building anything from simple websites to complex, high-traffic web applications. No matter your experience level, there is a Python framework that can help you bring your web project to life efficiently and effectively.

 

Python Web Development Frameworks

 

1. Django: The Full-Stack Powerhouse for Scalable Web Apps

Django is a robust, open-source Python framework designed for rapid development of secure and scalable web applications. With its built-in ORM, admin interface, authentication, and a vast ecosystem of reusable components, Django is ideal for building everything from simple websites to complex enterprise solutions. 

Learn more: https://www.djangoproject.com/

 

2. Flask: The Lightweight and Flexible Microframework

Flask is a minimalist Python web framework that gives you the essentials to get started, while letting you add only what you need. It’s perfect for small to medium-sized applications, APIs, and rapid prototyping. Flask’s simplicity, flexibility, and extensive documentation make it a top choice for developers who want full control over their project’s architecture.

Learn more: https://flask.palletsprojects.com/

 

3. FastAPI: Modern, High-Performance APIs with Ease

FastAPI is best known for building high-performance APIs, but with Jinja templates (v2), you can also create fully-featured websites that combine both backend and frontend functionality within the same framework. Built on top of Starlette and Pydantic, FastAPI offers asynchronous support, automatic interactive documentation, and exceptional speed, making it one of the fastest Python web frameworks available.

Learn more: https://fastapi.tiangolo.com/

 

4. Gradio: Effortless Web Interfaces for Machine Learning

Gradio is an open-source Python framework that allows you to rapidly build and share web-based interfaces for machine learning models. It is highly popular among the machine learning community, as you can build, test, and deploy your ML web demos on Hugging Face for free in just minutes. You don’t need front-end or back-end experience; just basic Python knowledge is enough to create high-performance web demos and APIs.

Learn more: https://www.gradio.app/

 

5. Streamlit: Instantly Build Data Web Apps

Streamlit is designed for data scientists and engineers who want to create beautiful, interactive web apps directly from Python scripts. With its intuitive API, you can build dashboards, data visualizations, and ML model demos in minutes.No need for HTML, CSS, or JavaScript. Streamlit is perfect for rapid prototyping and sharing insights with stakeholders.

Learn more: https://streamlit.io/

 

6. Tornado: Scalable, Non-Blocking Web Server and Framework

Tornado is a powerful Python web framework and asynchronous networking library, designed for building scalable and high-performance web applications. Unlike traditional frameworks, Tornado uses a non-blocking network I/O, which makes it ideal for handling thousands of simultaneous connections, perfect for real-time web services like chat applications, live updates, and long polling.

Learn more: https://www.tornadoweb.org/en/stable/guide.html 

 

7. Reflex: Pure Python Web Apps, Simplified

Reflex (formerly Pynecone) lets you build full-stack web applications using only Python, no JavaScript required. It compiles your Python code into modern web apps, handling both the frontend and backend seamlessly. Reflex is perfect for Python developers who want to create interactive, production-ready web apps without switching languages.

Learn more: https://reflex.dev/ 

 

Conclusion

 
FastAPI is my go-to framework for creating REST API endpoints for machine learning applications, thanks to its speed, simplicity, and production-ready features. 

For sharing machine learning demos with non-technical stakeholders, Gradio is incredibly useful, allowing you to build interactive web interfaces with minimal effort.

Django stands out as a robust, full-featured framework that lets you build any web-related application with complete control and scalability.

If you need something lightweight and quick to set up, Flask is an excellent choice for simple web apps and prototype.

Streamlit shines when it comes to building interactive user interfaces for data apps in just minutes, making it perfect for rapid prototyping and visualization.

For real-time web applications that require handling thousands of simultaneous connections, Tornado is a strong option due to its non-blocking, asynchronous architecture.

Finally, Reflex is a modern framework designed for building production-ready applications that are both simple to develop and easy to deploy.
 
 

Abid Ali Awan (@1abidaliawan) is a certified data scientist professional who loves building machine learning models. Currently, he is focusing on content creation and writing technical blogs on machine learning and data science technologies. Abid holds a Master’s degree in technology management and a bachelor’s degree in telecommunication engineering. His vision is to build an AI product using a graph neural network for students struggling with mental illness.



Source link

Continue Reading

Trending

Copyright © 2025 AISTORIZ. For enquiries email at prompt@travelstoriz.com