实战Spark分布式SQL引擎

一、概览
Spark SQL除了使用spark-sql命令进入交互式执行环境之外,还能够使用JDBC/ODBC或命令行接口进行分布式查询,在这个模式下,终端用户或应用可以直接和Spark SQL进行交互式SQL查询而不需要写任何scala代码。

二、使用Thrift JDBC server

spark版本    :1.4.0

Yarn版本    :CDH5.4.0

1、准备工作

将hive-site.xml拷贝或link到$SPARK_HOME/conf下

2、使用spark安装目录下脚本启动hive thrift server,默认不加参数时,会以local模式启动,占用本地一个JVM进程

sbin/start-thriftserver.sh

3、yarn-client模式启动,默认启动在10001端口

sbin/start-thriftserver.sh --master yarn

接下来,我们观察yarn UI的UI上,启动了25个container

实战Spark分布式SQL引擎

为什么启动了一个JDBC服务就占用这么多资源呢?这是因为conf/spark-env.sh中配置了SPARK_EXECUTOR_INSTANCES为24个实例,再加上一个yarn client的driver实例

export SPARK_EXECUTOR_INSTANCES=24

观察Yarn NodeManager节点上的进程,thriftserver会常驻一个叫org.apache.spark.executor.CoarseGrainedExecutorBackend的进程,随时为之后的SQL作业启动Task。这样做的好处是运行Spark SQL时,减少了启动container上的时间消耗,同时代价是在thrift server空闲的时候,这些container资源仍然占用着不会释放给其他spark或mapreduce作业使用。

实战Spark分布式SQL引擎

4、使用beeline连接Spark SQL交互式引擎

bin/beeline -u jdbc:hive2://localhost:10001 -n root -p root

注意,在非安全Hadoop模式下,用户名使用当前系统用户,密码为空或随意传值都可以;在kerberos Hadoop模式下,需要传递有效的principal令牌才可以登录beeline。

三、命令行帮助

1、Thrift server

Mandatory arguments to long options are mandatory for short options too.
  -a, --all                  do not ignore entries starting with .
  -A, --almost-all          do not list implied . and ..
      --author              with -l, print the author of each file
  -b, --escape              print octal escapes for nongraphic characters
      --block-size=SIZE      use SIZE-byte blocks.  See SIZE format below
  -B, --ignore-backups      do not list implied entries ending with ~
  -c                        with -lt: sort by, and show, ctime (time of last
                              modification of file status information)
                              with -l: show ctime and sort by name
                              otherwise: sort by ctime
  -C                        list entries by columns
      --color[=WHEN]        colorize the output.  WHEN defaults to `always'
                              or can be `never' or `auto'.  More info below
  -d, --directory            list directory entries instead of contents,
                              and do not dereference symbolic links
  -D, --dired                generate output designed for Emacs' dired mode
  -f                        do not sort, enable -aU, disable -ls --color
  -F, --classify            append indicator (one of */=>@|) to entries
      --file-type            likewise, except do not append `*'
      --format=WORD          across -x, commas -m, horizontal -x, long -l,
                              single-column -1, verbose -l, vertical -C
      --full-time            like -l --time-style=full-iso
  -g                        like -l, but do not list owner
      --group-directories-first
                            group directories before files.
                              augment with a --sort option, but any
                              use of --sort=none (-U) disables grouping
  -G, --no-group            in a long listing, don't print group names
  -h, --human-readable      with -l, print sizes in human readable format
                              (e.g., 1K 234M 2G)
      --si                  likewise, but use powers of 1000 not 1024
  -H, --dereference-command-line
                            follow symbolic links listed on the command line
      --dereference-command-line-symlink-to-dir
                            follow each command line symbolic link
                            that points to a directory
      --hide=PATTERN        do not list implied entries matching shell PATTERN
                              (overridden by -a or -A)
      --indicator-style=WORD  append indicator with style WORD to entry names:
                              none (default), slash (-p),
                              file-type (--file-type), classify (-F)
  -i, --inode                print the index number of each file
  -I, --ignore=PATTERN      do not list implied entries matching shell PATTERN
  -k                        like --block-size=1K
  -l                        use a long listing format
  -L, --dereference          when showing file information for a symbolic
                              link, show information for the file the link
                              references rather than for the link itself
  -m                        fill width with a comma separated list of entries
  -n, --numeric-uid-gid      like -l, but list numeric user and group IDs
  -N, --literal              print raw entry names (don't treat e.g. control
                              characters specially)
  -o                        like -l, but do not list group information
  -p, --indicator-style=slash
                            append / indicator to directories
  -q, --hide-control-chars  print ? instead of non graphic characters
      --show-control-chars  show non graphic characters as-is (default
                            unless program is `ls' and output is a terminal)
  -Q, --quote-name          enclose entry names in double quotes
      --quoting-style=WORD  use quoting style WORD for entry names:
                              literal, locale, shell, shell-always, c, escape
  -r, --reverse              reverse order while sorting
  -R, --recursive            list subdirectories recursively
  -s, --size                print the allocated size of each file, in blocks
  -S                        sort by file size
      --sort=WORD            sort by WORD instead of name: none -U,
                            extension -X, size -S, time -t, version -v
      --time=WORD            with -l, show time as WORD instead of modification
                            time: atime -u, access -u, use -u, ctime -c,
                            or status -c; use specified time as sort key
                            if --sort=time
      --time-style=STYLE    with -l, show times using style STYLE:
                            full-iso, long-iso, iso, locale, +FORMAT.
                            FORMAT is interpreted like `date'; if FORMAT is
                            FORMAT1<newline>FORMAT2, FORMAT1 applies to
                            non-recent files and FORMAT2 to recent files;
                            if STYLE is prefixed with `posix-', STYLE
                            takes effect only outside the POSIX locale
  -t                        sort by modification time
  -T, --tabsize=COLS        assume tab stops at each COLS instead of 8
  -u                        with -lt: sort by, and show, access time
                              with -l: show access time and sort by name
                              otherwise: sort by access time
  -U                        do not sort; list entries in directory order
  -v                        natural sort of (version) numbers within text
  -w, --width=COLS          assume screen width instead of current value
  -x                        list entries by lines instead of by columns
  -X                        sort alphabetically by entry extension
  -1                        list one file per line
 
SELinux options:
 
  --lcontext                Display security context.  Enable -l. Lines
                            will probably be too wide for most displays.
  -Z, --context              Display security context so it fits on most
                            displays.  Displays only mode, user, group,
                            security context and file name.
  --scontext                Display only security context and file name.
      --help    display this help and exit
      --version  output version information and exit

2、beeline

内容版权声明:除非注明,否则皆为本站原创文章。

转载注明出处:https://www.heiqu.com/79b9b2d6352be77554d43347503892d6.html