jupyterlab and pyspark2 integration in 1 minute

As we use CDH 5.14.0 on our hadoop cluster, the highest spark version to be support is 2.1.3, so this blog is to record the procedure of how I install pyspark-2.1.3 and integrate it with jupyter-lab.

Evironment:
spark 2.1.3
CDH 5.14.0 – hive 1.1.0
Anaconda3 – python 3.6.8

  1. Add export to spark-env.sh
    export PYSPARK_PYTHON=/opt/anaconda3/bin/python
    export PYSPARK_DRIVER_PYTHON=/opt/anaconda3/bin/jupyter-lab
    export PYSPARK_DRIVER_PYTHON_OPTS='  --ip=172.16.191.30 --port=8890'
  2. install sparkmagic
    pip install sparkmagic
  3. Use conda or pip command to downgrade ipykernel to 4.9.0, cause ipykernel 5.x doesn’t support sparkmagic, it will throw a Future exception.
    https://github.com/jupyter-incubator/sparkmagic/issues/492
  4. /opt/spark-2.1.3/bin/pyspark –master yarn

If you need to run with backgrand , use nohup.

Another problem, in pyspark, sqlContext cannot access remote hivemetastore and without any exceptions, when i run show databases in pyspark, it always return me default. And then i found out, in spark2’s jars dir, there was a hive-exec-1.1.0-cdh5.14.0.jar, delete this jar file, everythings ok.

自己动手打造ipv6梯子

以下内容适合有一定网络及梯子搭建经验的人士阅读。

准备工作:
Vultr美服主机一台,以下简称HAA,创建时选择同时支持IPv6和v4,最低价到20181212为止是3.5刀,选择位置靠近西海岸,Seattle,LA,均可,速度会比较好,自家网络测试觉得硅谷速度一般。

Vultr 欧服Paris,Amsterdam,Frankfurt均可,数量自定,选择IPv6 only,价格2.5刀。或者法国的scaleway,最低价格1.99欧。以下简称SSs。

阿里云或其他云主机一台,主要是利用其接入骨干网优势,可选操作,创建选择按量付费,最低配。

开工:
美服HAA部署HAProxy,配置如下

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3

defaults
        log     global
        mode    tcp # tcp 四层代理(默认http为七层代理)
        option  redispatch
        option  abortonclose
        timeout connect 6000
        timeout client  60000
        timeout server  60000
        timeout check   2000

listen  admin_stats
        bind    *:1090 #状态页,绑定ipv6[::]及ipv4 0.0.0.0 1090端口
        mode    http
        maxconn 10
        stats   refresh 30s
        stats   uri     /
        stats   realm   HAProxy
        stats   auth    haproxy:haproxy

listen ss
        bind    *:8388 #同时绑定ipv6[::]及v4 0.0.0.0地址 8388端口
        mode    tcp
        balance leastconn
        maxconn 8192
        server  Amsterdam-vultr                                         2001:19f0:5001:27ce:xxxx:xxx:xxxx:xxxx:8388      check   inter   180000  rise    1       fall    2 #反代阿姆斯特丹主机的ipv6地址的8388端口
        server  Amsterdam-scaleway                                      2001:bc8:xxxx:xxxx::1:8388                       check   inter   180000  rise    1       fall    2
        server  Seattle-vultr                                           2001:19f0:8001:1396:xxxx:xxx:xxxx:xxxx:8388      check   inter   180000  rise    1       fall    2

然后在对应的SSs云主机上部署和搭建ss server,这个教程很多,SSs配置时,IP绑定于::,例如:

{
  "server":"::",
  "port_password":{
    "995":"xxxxxxxxxx"
  },
  "timeout":300,
  "method":"aes-256-cfb",
  "workers": 10
}

这里需要注意的是,SS我看到的,Python版支持ipv6绑定,C版的貌似不支持IPv6绑定,没做更深入测试,所以这里我用的是Python版SS

启动SS并启动HAProxy,这就可以用了。好处是,仅IPv6的机器价格更低,被GFW拦截的可能性也更低,速度很快,且全球布局,连接速度主要取决于家里的连接美服的速度。如果资金不紧张,可以再搞一个国内的阿里云主机,最低配,搭建HAProxy再次反代美服的HAA,利用其骨干网优势。我家里是100Mbps宽带,通过阿云反代美服,上油土鳖等视频网站,带宽峰值可以到2-3MB,4K高清毫无压力。

所用服务器均为最低配,ubuntu18.04或debian stretch。

Enable HTTPS access in Zeppelin

I was using certified key file to enable HTTPS, if you use self-signatured key, see second part

First part:
I had got two files which one is  the private key named server.key and another one is certification file named server.crt
Use the following command to create a jks keystore file

openssl pkcs12 -export -in xxx.com.crt -inkey xxx.com.key -out xxx.com.pkcs12
keytool -importkeystore -srckeystore xxx.com.pkcs12 -destkeystore xxx.com.jks -srcstoretype pkcs12

Second part:
Use self-signatured key

# Generate root key file and cert file, key file could be named key or pem, it's same.
openssl genrsa -out root.key(pem) 2048 # Generate root key file
openssl req -x509 -new -key root.key(pem) -out root.crt # Generate root cert file

# Generate client key and cert and csr file
openssl genrsa -out client.key(pem) 2048 # Generate client key file
openssl req -new -key client.key(pem) -out client.csr # Generate client cert request file
openssl x509 -req -in client.csr -CA root.crt -CAkey root.key(pem) -CAcreateserial -days 3650 -out client.crt # Use root cert to generate client cert file

# Generate server key and cert and csr file
openssl genrsa -out server.key(pem) 2048 # Generate server key file, use in Zeppelin
openssl req -new -key server.key(pem) out server.csr @ Generate server cert request file
openssl x509 -req -in server.csr -CA root.crt -CAkey root.key(pem) -CAcreateserial -days 3650 -out server.crt # Use root cert to generate server cert file

# Generate client jks file
openssl pkcs12 -export -in client.crt -inkey client.key(pem) -out client.pkcs12 # Package to pkcs12 format, must input a password, you should remember the password
keytool -importkeystore -srckeystore client.pkcs12 -destkeystore client.jks -srcstoretype pkcs12 # The client password you just input at last step

# Generate server jks file
openssl pkcs12 -export -in server.crt -inkey server.key(pem) -out server.pkcs12 # Package to pkcs12 format, must input a password, you should remember the password
keytool -importkeystore -srckeystore server.pkcs12 -destkeystore server.jks -srcstoretype pkcs12 # The server password you just input at last step

The server key, cert and jks are using to configure zeppelin, the client key, cert and jks are using to install into browser or your client access codes.
Then, make a directory to put the server things in it, such as

mkdir -p /etc/zeppelin/conf/ssl
cp server.crt server.jks /etc/zeppelin/conf/ssl

And then modify zeppelin-site.xml to enable https access

<property>
  <name>zeppelin.server.ssl.port</name>
  <value>8443</value>
  <description>Server ssl port. (used when ssl property is set to true)</description>
</property>
<property>
  <name>zeppelin.ssl</name>
  <value>true</value>
  <description>Should SSL be used by the servers?</description>
</property>
<property>
  <name>zeppelin.ssl.client.auth</name>
  <value>false</value>
  <description>Should client authentication be used for SSL connections?</description>
</property>
<property>
  <name>zeppelin.ssl.keystore.path</name>
  <value>/etc/zeppelin/conf/ssl/xxx.com.jks</value>
  <description>Path to keystore relative to Zeppelin configuration directory</description>
</property>
<property>
  <name>zeppelin.ssl.keystore.type</name>
  <value>JKS</value>
  <description>The format of the given keystore (e.g. JKS or PKCS12)</description>
</property>
<property>
  <name>zeppelin.ssl.keystore.password</name>
  <value>password which you input on generating server jks step</value>
  <description>Keystore password. Can be obfuscated by the Jetty Password tool</description>
</property>

Then, all completed, and you can redirect 443 to 8443 by using iptables or other reverse proxy tools

试用时间序列数据库InfluxDB

Hadoop集群监控需要使用时间序列数据库,今天花了半天时间调研使用了一下最近比较火的InfluxDB,发现还真是不错,记录一下学习心得。

Influx是用Go语言写的,专为时间序列数据持久化所开发的,由于使用Go语言,所以各平台基本都支持。类似的时间序列数据库还有OpenTSDB,Prometheus等。 Continue reading 试用时间序列数据库InfluxDB

Tornado学习笔记(四)

一、Tornado的语言国际化方法

Tornado做国际化折腾了一下下,Tornado这部分的官方文档太poor了。所以自己记录一下如何用tornado结合gettext做国际化。

第一步,在项目路径下建立./locales/zh_CN/LC_MESSAGES文件夹。

第二步,使用xgettext或poedit在第一步的文件夹下创建一个po文件,比如messages.po,我用poedit创建,比xgettext方便一些。 Continue reading Tornado学习笔记(四)

Tornado学习笔记(一)

最近开始用Tornado做开发了,究其原因,主要是Tornado基于Python,一来代码量少开发速度快,二来采用epoll方式,能够承载的并发量很高。在我的i5台式机上用ab测试,不连接数据库的情况下,单用get生成页面,大概平均的并发量在7900左右。这比php或者java能够承载并发量都高很多很多。三来Python代码可维护性相对来说比php好很多,语法结构清晰。四来,tornado的框架设计的很黄很暴力,以HTTP请求方式作为方法名称,通常情况下,用户写一个页面只需要有get和post两种方式的方法定义就够了。 Continue reading Tornado学习笔记(一)