sql 求助,果然只会 mysql 的写 hive 要命

2022-05-16 20:36:33 +08:00
 Distand

数据开发跑路了,只能自己去 hive 导数据,需求是:

  1. xfrom='sugg'的全要

  2. 其它 xfrom 的按 sub_region 分组根据 pubtime 一个月前取 xx 条,一个月内取 xx 条,都要按 click 倒序排

  3. 汇总到一起

渣渣业务开发只能写出这种 sql 了,跑了下执行一次要 49 块钱,求个优化🙏

WITH total AS (
	SELECT * FROM xx.xx WHERE log_date='20220515'
),
other AS (
	SELECT * FROM total WHERE xfrom != 'sugg'
),
final AS (
  SELECT * FROM total WHERE xfrom='sugg' 
  UNION ALL
  SELECT * FROM (SELECT * FROM other WHERE sub_region='a' AND pubtime < '${yyyyMMdd,-1m}' ORDER BY click DESC LIMIT 1000) a1
  UNION ALL 
  SELECT * FROM (SELECT * FROM other WHERE sub_region='a' AND pubtime >= '${yyyyMMdd,-1m}' ORDER BY click DESC LIMIT 1000) a2
  UNION ALL 
  SELECT * FROM (SELECT * FROM other WHERE sub_region='b' AND pubtime < '${yyyyMMdd,-1m}' ORDER BY click DESC LIMIT 1000) b1
  UNION ALL 
  SELECT * FROM (SELECT * FROM other WHERE sub_region='b' AND pubtime >= '${yyyyMMdd,-1m}' ORDER BY click DESC LIMIT 1000) b2
  UNION ALL 
  SELECT * FROM (SELECT * FROM other WHERE sub_region='c' AND pubtime < '${yyyyMMdd,-1m}' ORDER BY click DESC LIMIT 250) c1
  UNION ALL 
  SELECT * FROM (SELECT * FROM other WHERE sub_region='c' AND pubtime >= '${yyyyMMdd,-1m}' ORDER BY click DESC LIMIT 250) c2
  UNION ALL 
  SELECT * FROM (SELECT * FROM other WHERE sub_region='d' AND pubtime < '${yyyyMMdd,-1m}' ORDER BY click DESC LIMIT 500) d1
  UNION ALL 
  SELECT * FROM (SELECT * FROM other WHERE sub_region='d' AND pubtime >= '${yyyyMMdd,-1m}' ORDER BY click DESC LIMIT 500) d2
  UNION ALL 
  SELECT * FROM (SELECT * FROM other WHERE sub_region='e' AND pubtime < '${yyyyMMdd,-1m}' ORDER BY click DESC LIMIT 500) e1
  UNION ALL 
  SELECT * FROM (SELECT * FROM other WHERE sub_region='e' AND pubtime >= '${yyyyMMdd,-1m}' ORDER BY click DESC LIMIT 500) e2
  UNION ALL 
  SELECT * FROM (SELECT * FROM other WHERE sub_region='f' AND pubtime < '${yyyyMMdd,-1m}' ORDER BY click DESC LIMIT 50) f1
  UNION ALL 
  SELECT * FROM (SELECT * FROM other WHERE sub_region='f' AND pubtime >= '${yyyyMMdd,-1m}' ORDER BY click DESC LIMIT 50) f2
  UNION ALL 
  SELECT * FROM (SELECT * FROM other WHERE sub_region='g' AND pubtime < '${yyyyMMdd,-1m}' ORDER BY click DESC LIMIT 50) g1
  UNION ALL 
  SELECT * FROM (SELECT * FROM other WHERE sub_region='g' AND pubtime >= '${yyyyMMdd,-1m}' ORDER BY click DESC LIMIT 50) g2
  UNION ALL 
  -- 这个 sub_region 有特殊要求,其它都是一样的条件,只是 limit 数量不同
  SELECT * FROM (SELECT * FROM other WHERE sub_region='h' AND pubtime < '${yyyyMMdd,-1m}' AND (tag LIKE '%th%' OR tag LIKE '%ph%') ORDER BY click DESC LIMIT 150) h1
  UNION ALL 
  SELECT * FROM (SELECT * FROM other WHERE sub_region='h' AND pubtime >= '${yyyyMMdd,-1m}' AND (tag LIKE '%th%' OR tag LIKE '%ph%') ORDER BY click DESC LIMIT 150) h2
)
INSERT OVERWRITE TABLE yy.yy SELECT * FROM final;
2004 次点击
所在节点    程序员
9 条回复
efaun
2022-05-16 21:07:32 +08:00
看到第二条说什么分组什么取多少条, 你看看分区函数适不适合你
https://www.cnblogs.com/linJie1930906722/p/6036053.html
chineselittleboy
2022-05-16 21:55:15 +08:00
最里面的 select * 改成想要的一些字段能便宜点
zhusimaji
2022-05-16 23:25:58 +08:00
关于取数分组的语句可以参考 select * from (select XXX, row_number() over (partition by sub_region order by click desc ) as rank from XXX)a where rank <1000
zhusimaji
2022-05-16 23:26:46 +08:00
所以后续一个月内和一个月外只要 union 一下就可以了,应该就可以输出结果了
FYFX
2022-05-16 23:30:44 +08:00
你下面那一堆 sql 可以根据 pubtime 分成两部分,然后用窗口函数 partition by sub_region order by click 算一下就行了吧,而且你这只取一天数据要 49 块钱是不是有点离谱
lyang
2022-05-17 08:36:42 +08:00
49 块钱是什么意思
Distand
2022-05-17 09:48:25 +08:00
感谢各位老哥,先 case when pubtime < '${yyyyMMdd,-1m}' then 0 else 1 end as pt ,再 row_number() over(partition by sub_region,pt order by click desc) as rank 后只需要 2 块钱了
512357301
2022-05-17 13:15:21 +08:00
这是一条 SQL 就搞定的吧(通过辅助列判断需要哪些数据,需要用到开窗函数,然后 where 就行了,怎么你这搞了那么多的 union all ,hive 执行 union all 效率很低的
shuianfendi6
2022-05-20 14:51:56 +08:00
用 partition 和 row_number 吧

spark-sql 执行效率会好不少

这是一个专为移动设备优化的页面(即为了让你能够在 Google 搜索结果里秒开这个页面),如果你希望参与 V2EX 社区的讨论,你可以继续到 V2EX 上打开本讨论主题的完整版本。

https://www.v2ex.com/t/853265

V2EX 是创意工作者们的社区,是一个分享自己正在做的有趣事物、交流想法,可以遇见新朋友甚至新机会的地方。

V2EX is a community of developers, designers and creative people.

© 2021 V2EX