Scroll

What to do if Cloud Discovery Scan does not show any results even if columns look fine?

Follow

While scanning some logs, there are no results to be seen even though the formats and column definitions look fine.  In such cases, it might be that the file may be in a different encoding.  This happens with Websense logs, but can happen with other logs too.  In some of cases, Websense logs are encoded with UTF-16 and Skyfence Cloud Discovery Tool supports only UTF-8 in current versions.  Support of other encoding methods will be available in later versions.

Convert the files to UTF8 first and then run discovery tool.  Easiest way to find out if Cloud discovery tool had a problem with encoding is to use data preview option within the tool and some columns which are mapped corrected are missing or all of the columns missing and data is distorted with spaces appearing between each character.

UTF-16 (Not Supported):

UTF-8 (Good):

 

Steps to convert from UTF-16 (or other encoding) to UTF-8:

Notepad (Default Windows text editor):

Open text file in UTF-16 (or other encoding) -> Save as -> Choose Encoding as UTF-8 -> Save

 

Convert multiple files in  Linux/Unix using "iconv" command:

Eg. to convert all files with extension *.tmp to utf8:

for file in *.tmp; do iconv -f UTF-16 -t UTF-8 "$file" -o "${file%.tmp}.utf8"; done

[root@labvm1 WebsenseLogs]# ls -al
total 16588
drwxr-xr-x 3 root root 4096 Sep 17 09:55 .
drwxrwxrwt 8 root root 4096 Sep 17 02:39 ..
-rw-r--r-- 1 root root 1154432 Jul 8 2013 log212D_tritonlog-prd1_070813.tmp
-rw-r--r-- 1 root root 3145904 Jul 8 2013 log4271_tritonlog-prd1_070813.tmp
-rw-r--r-- 1 root root 3146126 Jul 8 2013 log43CF_tritonlog-prd1_070813.tmp
-rw-r--r-- 1 root root 210534 Jul 8 2013 log57AF_tritonlog-prd1_070813.tmp
-rw-r--r-- 1 root root 1597544 Jul 8 2013 log585E_tritonlog-prd1_070813.tmp
-rw-r--r-- 1 root root 858092 Jul 8 2013 log59EA_tritonlog-prd1_070813.tmp
-rw-r--r-- 1 root root 22170 Jul 8 2013 log63B4_tritonlog-prd1_070813.tmp
-rw-r--r-- 1 root root 3146052 Jul 8 2013 log8BA1_tritonlog-prd1_070813.tmp
drwxr-xr-x 2 root root 4096 Sep 17 09:54 utf8
-rw-r--r-- 1 root root 1610855 Sep 17 09:55 utf8.tgz
-rw-r--r-- 1 root root 2050011 Sep 17 02:06 WebsenseLogs.zip
-rw-r--r-- 1 root root 654 Jul 8 2013 wtg7710_tritonlog-prd1_070813.tmp
[root@labvm1 WebsenseLogs]#
[root@labvm1 WebsenseLogs]#
[root@labvm1 WebsenseLogs]#
[root@labvm1 WebsenseLogs]# for file in *.tmp; do iconv -f UTF-16 -t UTF-8 "$file" -o "${file%.tmp}.utf8"; done

Converted files:

[root@labvm1 WebsenseLogs]# ls -al utf8
total 6516
drwxr-xr-x 2 root root 4096 Sep 17 09:54 .
drwxr-xr-x 3 root root 4096 Sep 17 09:55 ..
-rw-r--r-- 1 root root 577216 Sep 17 09:53 log212D_tritonlog-prd1_070813.utf8
-rw-r--r-- 1 root root 1572953 Sep 17 09:53 log4271_tritonlog-prd1_070813.utf8
-rw-r--r-- 1 root root 1573063 Sep 17 09:53 log43CF_tritonlog-prd1_070813.utf8
-rw-r--r-- 1 root root 105267 Sep 17 09:53 log57AF_tritonlog-prd1_070813.utf8
-rw-r--r-- 1 root root 798772 Sep 17 09:53 log585E_tritonlog-prd1_070813.utf8
-rw-r--r-- 1 root root 429046 Sep 17 09:53 log59EA_tritonlog-prd1_070813.utf8
-rw-r--r-- 1 root root 11085 Sep 17 09:53 log63B4_tritonlog-prd1_070813.utf8
-rw-r--r-- 1 root root 1573026 Sep 17 09:53 log8BA1_tritonlog-prd1_070813.utf8
-rw-r--r-- 1 root root 327 Sep 17 09:53 wtg7710_tritonlog-prd1_070813.utf8



Have more questions? Submit a request

Comments

Powered by Zendesk