Show HN: SQL 約25行からFlashAttentionカーネルを自動推論
Show HN: Automatic deduction of FlashAttention kernel from ~25 lines of SQL (github.com)
要約
この記事では、SQL約25行でFlashAttentionカーネルを自動推論するデモを紹介しています。これは、クラウド上のペタバイト級配列データセットを効率的にテーブルとして表現するxarray-sql、標準SQLに自動微分を追加するddx、高速テンソル縮約のためのeifoldという3つのポータブルSQL拡張機能によって可能になります。このアプローチは、テンソルプログラムの論理層と物理層を分離することで、アクセラレータプログラミングに新たな視点をもたらすことを目指しています。
全文翻訳
Show HN: Automatic deduction of FlashAttention kernel from ~25 lines of SQL
In this demo, we automatically deduce a FlashAttention kernel with ~25 lines of SQL, which just expresses vanilla attention. This is possible due to three portable SQL extensions that we’re building: - xarray-sql: a library that efficiently represents petabyte scale array datasets in the cloud as tables (relations). - ddx: a database extension that adds automatic differentiation in the style of JAX to standard SQL, assuming the above array data model. - einfold: a modular query planner for fast tensor contractions in vanilla SQL, again, assuming the above array data model.
My hope is that the relational model and algebra provide fresh perspectives on how we might do accelerator programming. Here, our thesis is that separating the logical layers in a tensor program from the physical layers is both more correct and more scalable.
If the Bitter Lesson is a central dogma of machine learning, then why not formalize it with relational algebra?
If this interests you and you’d like to tinker with us at the intersection of database theory and ML systems, join us on Discord at https://xql.systems (link at the bottom). Cheers.