I always wish I wouldn’t have to pull in python/pandas to do simple data transformations on smaller local datasets/csv files and stick with sqlite. But then I tend to quickly hit a problem that seems to require dynamic SQL. Anything in between SQL and a general programming language that is made for transforming tabular data?
You must log in or register to comment.
Can I suggest duckdb? You can start out writing SQL directly on top of CSV and Parquet files. But then if you want/need to do something more complicated, you can
import duckdbinto Python, keep running the SQL you already wrote, convert it to a Pandas or Polars dataframe, transform it, then query the result in SQL.