最新消息:雨落星辰是一个专注网站SEO优化、网站SEO诊断、搜索引擎研究、网络营销推广、网站策划运营及站长类的自媒体原创博客

Empty SAS Studio Error when using large Work Tables imported from PostgreSQL DB - Stack Overflow

programmeradmin5浏览0评论

This might be related to this question, but in my current SAS Viya Stable 2024.12 environment I encounter an empty Error Message and the SAS Studio session context breaks, while trying to do a SELECT DISTINCT in a PROC SQL step.


My table was loaded from postgres DB to WORK with SELECT *. It has 1.18 Mio. rows, Col1 & Col3 are of type varchar(64), Col2 is of type varchar(256) the other 27 columns are not of interest. (The exported .sas7bdat file from work is like 283 MB)


If I run

PROC SQL _method;
   CREATE TABLE WORK.OUTPUT AS SELECT
      t1.col1, t1.col2, t1.col3
   FROM WORK.INPUT AS t1;
QUIT;

it works like a charm and needs like 3 seconds, with following log:

NOTE: SQL execution methods chosen are:
      sqxcrta
          sqxsrc( WORK.INPUT(alias = T1) )
NOTE: Compressing data set WORK.OUTPUT decreased size by 95.76 percent. 
      Compressed is 1,192 pages; un-compressed would require 28,144 pages.
NOTE: Table WORK.OUTPUT created, with 1,182,028 rows and 3 columns.
NOTE: PROZEDUR SQL used (Total process time):
      real time           1.55 Seconds
      user cpu time       1.47 Seconds
      system cpu time     0.12 Seconds
      memory               6,032.03k
      OS Memory           30,448.00k
      Timestamp           ...
      Step Count                        12  Switch Count  0
      Page Faults                             0
      Page Reclaims                         244
      Page Swaps                              0
      Voluntary Context Switches             29
      Involuntary Context Switches            8
      Block Input Operations                  0
      Block Output Operations           153,360

But if I use "SELECT DISTINCT" it raises an empty error and the log stops at:

NOTE: SQL execution methods chosen are:
      sqxcrta
          sqxunqs
              sqxsrc( WORK.INPUT(alias = T1) )

This also happen if I only SELECT DISTINCT on t1.Col2 (or a combination of four other varchar(64) columns...).


Apparently, the loaded varchar columns have quadrupled in size (256 → 1024), and I encountered the following behavior:

  • SELECT DISTINCT t1.col2 - Fails
  • SELECT DISTINCT PUT(t1.col2, %256.) - Succeeds
  • SELECT DISTINCT PUT(t1.col2, %1024.) - Fails

I tried to monitor and raise the memory usage, as follows, as well as executing on an older preliminary env with Viya Stable 2024.08, and there it works:

STATUS Distinct MEMSIZE MAXMEMQUERY
PreEnv Viya Succeeds 2,147,483,648 268,435,456
Current Viya Dev Fails 4,294,967,296 268,435,456
Current Viya Test Fails 21,474,836,480 268,435,456
发布评论

评论列表(0)

  1. 暂无评论