Python 官方文档:入门教程 => 点击学习
ubuntu自带的python 版本是2.7, 我们要把pyspark默认改成anaconda Python 3.6 down vot You can specify the version of Python for the drive
我们要把pyspark默认改成anaconda Python 3.6
down vot
You can specify the version of Python for the driver by setting the appropriate environment variables in the ./conf/spark-env.sh file. If it doesn't already exist, you can use the spark-env.sh.templatefile provided which also includes lots of other variables.
Here is a simple example of a spark-env.sh file to set the relevant Python environment variables:
#!/usr/bin/env bash
# This file is sourced when running various Spark programs.
export PYSPARK_PYTHON=/usr/bin/python3
export PYSPARK_DRIVER_PYTHON=/usr/bin/ipython
In this case it sets the version of Python used by the workers/executors to Python3 and the driver version of Python to iPython for a nicer shell to work in.
意思就是把spark文件夹下的./conf/spark-env.sh.tempalte 重命名成spark-env.sh
然后添加如下内容:
# This file is sourced when running various Spark programs.
export PYSPARK_PYTHON=/usr/bin/python3
export PYSPARK_DRIVER_PYTHON=/usr/bin/ipython
重启spark 即可
--结束END--
本文标题: pyspark修改python版本
本文链接: https://www.lsjlt.com/news/184776.html(转载时请注明来源链接)
有问题或投稿请发送至: 邮箱/279061341@qq.com QQ/279061341
下载Word文档到电脑,方便收藏和打印~
2024-03-01
2024-03-01
2024-03-01
2024-02-29
2024-02-29
2024-02-29
2024-02-29
2024-02-29
2024-02-29
2024-02-29
回答
回答
回答
回答
回答
回答
回答
回答
回答
回答
0