Select, Filter, Aggregate, Assign (#221)

* add column expr * update * update * update select * add conditions * update * add select and filter * simplify * simplify * update engine docs * update * update workflow interfaces * fix spark alter columns * update * update * add aggregate to execution engine * add aggregate * fix coverage * update * update comments * update docs * update doc * update docs
fugue-project · Jun 16, 2021 · 23a91af · 23a91af
1 parent f1ff762
commit 23a91af
Show file tree

Hide file tree

Showing 41 changed files with 3,137 additions and 48 deletions.
diff --git a/.pylintrc b/.pylintrc
@@ -1,3 +1,3 @@
 [MESSAGES CONTROL]
-disable = C0103,C0114,C0115,C0116,C0122,C0200,C0302,C0411,C0415,E0401,E0712,R0201,R0205,R0801,R0902,R0903,R0904,R0911,R0912,R0913,R0914,R1705,R1710,R1720,R1724,W0102,W0107,W0108,W0201,W0212,W0221,W0223,W0511,W0613,W0631,W0640,W0703,W1116
+disable = C0103,C0114,C0115,C0116,C0122,C0200,C0302,C0411,C0415,E0401,E0712,E1130,R0201,R0205,R0801,R0902,R0903,R0904,R0911,R0912,R0913,R0914,R1705,R1710,R1718,R1720,R1724,W0102,W0107,W0108,W0201,W0212,W0221,W0223,W0511,W0613,W0631,W0640,W0703,W1116
 # TODO: R0205: inherits from object, can be safely removed
diff --git a/RELEASE.md b/RELEASE.md
@@ -1,7 +1,8 @@
 # Release Notes
 
-## 0.5.7
+## 0.6.0
 
+-   Added Select, Aggregate, Filter, Assign [interfaces](https://github.com/fugue-project/fugue/issues/211)
 -   Let SparkExecutionEngine accept [empty](https://github.com/fugue-project/fugue/issues/217) pandas dataframes
 -   Move to [codecov](https://github.com/fugue-project/fugue/issues/216)
 -   Let Fugue SQL take input dataframes with name such as [a.b](https://github.com/fugue-project/fugue/issues/215)

diff --git a/docs/_templates/package.rst_t b/docs/_templates/package.rst_t
@@ -45,7 +45,7 @@
 .. |FugueSQLTutorial| replace:: :ref:`the Fugue SQL Tutorial <tutorial:/tutorials/sql.ipynb>`
 .. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:/tutorials/schema_dataframes.ipynb#dataframe>`
 .. |ExecutionEngineTutorial| replace:: :ref:`the ExecutionEngine Tutorial <tutorial:/tutorials/execution_engine.ipynb>`
-
+.. |FugueDataTypes| replace:: :ref:`Fugue Data Types <tutorial:/tutorials/appendix/generate_types.ipynb>`
 
 {% if separatemodules %}
 {%- else %}

diff --git a/docs/api/fugue.collections.rst b/docs/api/fugue.collections.rst
@@ -18,7 +18,7 @@ fugue.collections
 .. |FugueSQLTutorial| replace:: :ref:`the Fugue SQL Tutorial <tutorial:/tutorials/sql.ipynb>`
 .. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:/tutorials/schema_dataframes.ipynb#dataframe>`
 .. |ExecutionEngineTutorial| replace:: :ref:`the ExecutionEngine Tutorial <tutorial:/tutorials/execution_engine.ipynb>`
-
+.. |FugueDataTypes| replace:: :ref:`Fugue Data Types <tutorial:/tutorials/appendix/generate_types.ipynb>`
 
 
 fugue.collections.partition

diff --git a/docs/api/fugue.column.rst b/docs/api/fugue.column.rst
@@ -0,0 +1,47 @@
+fugue.column 
+=============
+
+.. |SchemaLikeObject| replace:: :ref:`Schema like object <tutorial:/tutorials/x-like.ipynb#schema>`
+.. |ParamsLikeObject| replace:: :ref:`Parameters like object <tutorial:/tutorials/x-like.ipynb#parameters>`
+.. |DataFrameLikeObject| replace:: :ref:`DataFrame like object <tutorial:/tutorials/x-like.ipynb#dataframe>`
+.. |DataFramesLikeObject| replace:: :ref:`DataFrames like object <tutorial:/tutorials/x-like.ipynb#dataframes>`
+.. |PartitionLikeObject| replace:: :ref:`Partition like object <tutorial:/tutorials/x-like.ipynb#partition>`
+.. |RPCHandlerLikeObject| replace:: :ref:`RPChandler like object <tutorial:/tutorials/x-like.ipynb#rpc>`
+
+.. |ExecutionEngine| replace:: :class:`~fugue.execution.execution_engine.ExecutionEngine`
+.. |NativeExecutionEngine| replace:: :class:`~fugue.execution.native_execution_engine.NativeExecutionEngine`
+.. |FugueWorkflow| replace:: :class:`~fugue.workflow.workflow.FugueWorkflow`
+
+.. |ReadJoin| replace:: Read Join tutorials on :ref:`workflow <tutorial:/tutorials/dag.ipynb#join>` and :ref:`engine <tutorial:/tutorials/execution_engine.ipynb#join>` for details
+.. |FugueConfig| replace:: :ref:`the Fugue Configuration Tutorial <tutorial:/tutorials/useful_config.ipynb>`
+.. |PartitionTutorial| replace:: :ref:`the Partition Tutorial <tutorial:/tutorials/partition.ipynb>`
+.. |FugueSQLTutorial| replace:: :ref:`the Fugue SQL Tutorial <tutorial:/tutorials/sql.ipynb>`
+.. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:/tutorials/schema_dataframes.ipynb#dataframe>`
+.. |ExecutionEngineTutorial| replace:: :ref:`the ExecutionEngine Tutorial <tutorial:/tutorials/execution_engine.ipynb>`
+.. |FugueDataTypes| replace:: :ref:`Fugue Data Types <tutorial:/tutorials/appendix/generate_types.ipynb>`
+
+
+fugue.column.expressions
+------------------------
+
+.. automodule:: fugue.column.expressions
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+fugue.column.functions
+----------------------
+
+.. automodule:: fugue.column.functions
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+fugue.column.sql
+----------------
+
+.. automodule:: fugue.column.sql
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
diff --git a/docs/api/fugue.dataframe.rst b/docs/api/fugue.dataframe.rst
@@ -18,7 +18,7 @@ fugue.dataframe
 .. |FugueSQLTutorial| replace:: :ref:`the Fugue SQL Tutorial <tutorial:/tutorials/sql.ipynb>`
 .. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:/tutorials/schema_dataframes.ipynb#dataframe>`
 .. |ExecutionEngineTutorial| replace:: :ref:`the ExecutionEngine Tutorial <tutorial:/tutorials/execution_engine.ipynb>`
-
+.. |FugueDataTypes| replace:: :ref:`Fugue Data Types <tutorial:/tutorials/appendix/generate_types.ipynb>`
 
 
 fugue.dataframe.array\_dataframe

diff --git a/docs/api/fugue.execution.rst b/docs/api/fugue.execution.rst
@@ -18,7 +18,7 @@ fugue.execution
 .. |FugueSQLTutorial| replace:: :ref:`the Fugue SQL Tutorial <tutorial:/tutorials/sql.ipynb>`
 .. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:/tutorials/schema_dataframes.ipynb#dataframe>`
 .. |ExecutionEngineTutorial| replace:: :ref:`the ExecutionEngine Tutorial <tutorial:/tutorials/execution_engine.ipynb>`
-
+.. |FugueDataTypes| replace:: :ref:`Fugue Data Types <tutorial:/tutorials/appendix/generate_types.ipynb>`
 
 
 fugue.execution.execution\_engine

diff --git a/docs/api/fugue.extensions.creator.rst b/docs/api/fugue.extensions.creator.rst
@@ -18,7 +18,7 @@ fugue.extensions.creator
 .. |FugueSQLTutorial| replace:: :ref:`the Fugue SQL Tutorial <tutorial:/tutorials/sql.ipynb>`
 .. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:/tutorials/schema_dataframes.ipynb#dataframe>`
 .. |ExecutionEngineTutorial| replace:: :ref:`the ExecutionEngine Tutorial <tutorial:/tutorials/execution_engine.ipynb>`
-
+.. |FugueDataTypes| replace:: :ref:`Fugue Data Types <tutorial:/tutorials/appendix/generate_types.ipynb>`
 
 
 fugue.extensions.creator.convert

diff --git a/docs/api/fugue.extensions.outputter.rst b/docs/api/fugue.extensions.outputter.rst
@@ -18,7 +18,7 @@ fugue.extensions.outputter
 .. |FugueSQLTutorial| replace:: :ref:`the Fugue SQL Tutorial <tutorial:/tutorials/sql.ipynb>`
 .. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:/tutorials/schema_dataframes.ipynb#dataframe>`
 .. |ExecutionEngineTutorial| replace:: :ref:`the ExecutionEngine Tutorial <tutorial:/tutorials/execution_engine.ipynb>`
-
+.. |FugueDataTypes| replace:: :ref:`Fugue Data Types <tutorial:/tutorials/appendix/generate_types.ipynb>`
 
 
 fugue.extensions.outputter.convert

diff --git a/docs/api/fugue.extensions.processor.rst b/docs/api/fugue.extensions.processor.rst
@@ -18,7 +18,7 @@ fugue.extensions.processor
 .. |FugueSQLTutorial| replace:: :ref:`the Fugue SQL Tutorial <tutorial:/tutorials/sql.ipynb>`
 .. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:/tutorials/schema_dataframes.ipynb#dataframe>`
 .. |ExecutionEngineTutorial| replace:: :ref:`the ExecutionEngine Tutorial <tutorial:/tutorials/execution_engine.ipynb>`
-
+.. |FugueDataTypes| replace:: :ref:`Fugue Data Types <tutorial:/tutorials/appendix/generate_types.ipynb>`
 
 
 fugue.extensions.processor.convert

diff --git a/docs/api/fugue.extensions.rst b/docs/api/fugue.extensions.rst
@@ -26,7 +26,7 @@ fugue.extensions
 .. |FugueSQLTutorial| replace:: :ref:`the Fugue SQL Tutorial <tutorial:/tutorials/sql.ipynb>`
 .. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:/tutorials/schema_dataframes.ipynb#dataframe>`
 .. |ExecutionEngineTutorial| replace:: :ref:`the ExecutionEngine Tutorial <tutorial:/tutorials/execution_engine.ipynb>`
-
+.. |FugueDataTypes| replace:: :ref:`Fugue Data Types <tutorial:/tutorials/appendix/generate_types.ipynb>`
 
 
 fugue.extensions.context

diff --git a/docs/api/fugue.extensions.transformer.rst b/docs/api/fugue.extensions.transformer.rst
@@ -18,7 +18,7 @@ fugue.extensions.transformer
 .. |FugueSQLTutorial| replace:: :ref:`the Fugue SQL Tutorial <tutorial:/tutorials/sql.ipynb>`
 .. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:/tutorials/schema_dataframes.ipynb#dataframe>`
 .. |ExecutionEngineTutorial| replace:: :ref:`the ExecutionEngine Tutorial <tutorial:/tutorials/execution_engine.ipynb>`
-
+.. |FugueDataTypes| replace:: :ref:`Fugue Data Types <tutorial:/tutorials/appendix/generate_types.ipynb>`
 
 
 fugue.extensions.transformer.constants

diff --git a/docs/api/fugue.rpc.rst b/docs/api/fugue.rpc.rst
@@ -18,7 +18,7 @@ fugue.rpc
 .. |FugueSQLTutorial| replace:: :ref:`the Fugue SQL Tutorial <tutorial:/tutorials/sql.ipynb>`
 .. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:/tutorials/schema_dataframes.ipynb#dataframe>`
 .. |ExecutionEngineTutorial| replace:: :ref:`the ExecutionEngine Tutorial <tutorial:/tutorials/execution_engine.ipynb>`
-
+.. |FugueDataTypes| replace:: :ref:`Fugue Data Types <tutorial:/tutorials/appendix/generate_types.ipynb>`
 
 
 fugue.rpc.base

diff --git a/docs/api/fugue.rst b/docs/api/fugue.rst
@@ -5,6 +5,7 @@ fugue
    :maxdepth: 4
 
    fugue.collections
+   fugue.column
    fugue.dataframe
    fugue.execution
    fugue.extensions
@@ -28,7 +29,7 @@ fugue
 .. |FugueSQLTutorial| replace:: :ref:`the Fugue SQL Tutorial <tutorial:/tutorials/sql.ipynb>`
 .. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:/tutorials/schema_dataframes.ipynb#dataframe>`
 .. |ExecutionEngineTutorial| replace:: :ref:`the ExecutionEngine Tutorial <tutorial:/tutorials/execution_engine.ipynb>`
-
+.. |FugueDataTypes| replace:: :ref:`Fugue Data Types <tutorial:/tutorials/appendix/generate_types.ipynb>`
 
 
 fugue.constants

diff --git a/docs/api/fugue.workflow.rst b/docs/api/fugue.workflow.rst
@@ -18,7 +18,7 @@ fugue.workflow
 .. |FugueSQLTutorial| replace:: :ref:`the Fugue SQL Tutorial <tutorial:/tutorials/sql.ipynb>`
 .. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:/tutorials/schema_dataframes.ipynb#dataframe>`
 .. |ExecutionEngineTutorial| replace:: :ref:`the ExecutionEngine Tutorial <tutorial:/tutorials/execution_engine.ipynb>`
-
+.. |FugueDataTypes| replace:: :ref:`Fugue Data Types <tutorial:/tutorials/appendix/generate_types.ipynb>`
 
 
 fugue.workflow.module

diff --git a/docs/api_dask/fugue_dask.rst b/docs/api_dask/fugue_dask.rst
@@ -18,7 +18,7 @@ fugue\_dask
 .. |FugueSQLTutorial| replace:: :ref:`the Fugue SQL Tutorial <tutorial:/tutorials/sql.ipynb>`
 .. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:/tutorials/schema_dataframes.ipynb#dataframe>`
 .. |ExecutionEngineTutorial| replace:: :ref:`the ExecutionEngine Tutorial <tutorial:/tutorials/execution_engine.ipynb>`
-
+.. |FugueDataTypes| replace:: :ref:`Fugue Data Types <tutorial:/tutorials/appendix/generate_types.ipynb>`
 
 
 fugue\_dask.dataframe

diff --git a/docs/api_spark/fugue_spark.rst b/docs/api_spark/fugue_spark.rst
@@ -18,7 +18,7 @@ fugue\_spark
 .. |FugueSQLTutorial| replace:: :ref:`the Fugue SQL Tutorial <tutorial:/tutorials/sql.ipynb>`
 .. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:/tutorials/schema_dataframes.ipynb#dataframe>`
 .. |ExecutionEngineTutorial| replace:: :ref:`the ExecutionEngine Tutorial <tutorial:/tutorials/execution_engine.ipynb>`
-
+.. |FugueDataTypes| replace:: :ref:`Fugue Data Types <tutorial:/tutorials/appendix/generate_types.ipynb>`
 
 
 fugue\_spark.dataframe

diff --git a/docs/api_sql/fugue_sql.rst b/docs/api_sql/fugue_sql.rst
@@ -18,7 +18,7 @@ fugue\_sql
 .. |FugueSQLTutorial| replace:: :ref:`the Fugue SQL Tutorial <tutorial:/tutorials/sql.ipynb>`
 .. |DataFrameTutorial| replace:: :ref:`the DataFrame Tutorial <tutorial:/tutorials/schema_dataframes.ipynb#dataframe>`
 .. |ExecutionEngineTutorial| replace:: :ref:`the ExecutionEngine Tutorial <tutorial:/tutorials/execution_engine.ipynb>`
-
+.. |FugueDataTypes| replace:: :ref:`Fugue Data Types <tutorial:/tutorials/appendix/generate_types.ipynb>`
 
 
 fugue\_sql.exceptions

diff --git a/fugue/column/__init__.py b/fugue/column/__init__.py
@@ -0,0 +1,4 @@
+# flake8: noqa
+from fugue.column.expressions import ColumnExpr, col, function, lit, null
+from fugue.column.functions import is_agg
+from fugue.column.sql import SelectColumns, SQLExpressionGenerator