[MINOR][PYTHON][DOCS] Clarify verifySchema at createDataFrame not wor…

…king with pandas DataFrame with Arrow optimization ### What changes were proposed in this pull request? This PR proposes to clarify that `verifySchema` at createDataFrame does not wotj with pandas DataFrame with Arrow optimization enabled. ### Why are the changes needed? For correct information about `verifySchema` <> Arrow optimization in `createDataFrame`. ### Does this PR introduce _any_ user-facing change? Yes, it fixes the user-facing documentation. ### How was this patch tested? I manually ran linters ### Was this patch authored or co-authored using generative AI tooling? No. Closes #45333 from HyukjinKwon/improve-doc-verifySchema. Authored-by: Hyukjin Kwon <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
apache · Feb 29, 2024 · a8b6e3c · a8b6e3c
1 parent 11e8ae4
commit a8b6e3c
Showing 1 changed file with 3 additions and 0 deletions.
diff --git a/python/pyspark/sql/session.py b/python/pyspark/sql/session.py
@@ -1325,6 +1325,9 @@ def createDataFrame(  # type: ignore[misc]
             if ``samplingRatio`` is ``None``.
         verifySchema : bool, optional
             verify data types of every row against schema. Enabled by default.
+            When the input is :class:`pandas.DataFrame` and
+            `spark.sql.execution.arrow.pyspark.enabled` is enabled, this option is not
+            effective. It follows Arrow type coercion.
 
             .. versionadded:: 2.1.0