What is the optimum log file duration and fields required in order to get the most accurate discovery report?


Skyfence Cloud Discovery tool can analyze multiple log files from small to a very large in size.  There is no specific rule to have a specific size file, but it is always better to have a log file spanning at least a week to a month or more and consists of good distribution of user traffic within it. It is difficult to predict such distribution, but better to avoid log files obtained when many in the company is on an offsite meeting or conference.  

User traffic is usually contained in logs of security devices like firewall, proxy/web proxy, network security monitors such as ids, etc.  Hence it is recommended to work with network administrators who manage them to export traffic logs to .csv or table formatted files with specific fields which will be useful for analysis.

Skyfence Cloud Discovery tool can also be used to scan entire folder and its subfolders consisting of log files.  This option is useful to scan multiple days of logs and configure continuous discovery scans and analysis.

There are mandatory and optional fields to be configured within the parser(s) to extract relevant data to correlate users, their activities with each cloud application and analyze the logs effectively.

Please note that if any one of these mandatory fields are missing, then the tool can not analyze the data nor will result in partial/inaccurate analysis/report.

Mandatory Fields:

  • DATE - Date of the event
  • TIME - Time of the event
  • SOURCE - Source IP
  • DESTINATION - Destination host IP

Optional Fields:

  • TRAFFIC - Total traffic received and sent in bytes or Kb
  • HOSTNAME  - FQDN of website visited or entire Url
  • ACTION - Action taken by device if traffic was allowed or denied
  • USER - Application user who accessed cloud application or specific url

Each of above fields can also use standard regular expressions to extend the parser to address any pattern matching within column content.  For eg. DATE column can take any standard date formats such dd/MM/yyyy (05/10/2014) or dd MMM yyyy (05 May 2014).  Similarly, TIME columns can be defined with specific formats including support for unix timestamps.

Here are some of common log columns/fields which can be mapped to Skyfence Cloud Discovery parsers in devices mentioned above depending upon the make, model and vendor of such devices.

  • Receive Time
  • Serial #
  • Type
  • Threat/Content Type
  • Config Version 
  • Generate Time 
  • Source address 
  • Destination address 
  • NAT Source IP 
  • NAT Destination IP 
  • Rule 
  • Source User 
  • Destination User 
  • Application 
  • Virtual System 
  • Source Zone 
  • Destination Zone 
  • Inbound Interface 
  • Outbound Interface 
  • Log Action 
  • Time Logged 
  • Session ID 
  • Repeat Count 
  • Source Port 
  • Destination Port 
  • NAT Source Port 
  • NAT Destination Port 
  • Flags 
  • IP Protocol 
  • Action 
  • Bytes 
  • Bytes Sent 
  • Bytes Received 
  • Packets Start Time 
  • Elapsed Time (sec) 
  • Category 
  • Padding seqno 
  • actionflags 
  • Source Country 
  • Destination Country 
  • cpadding 
  • pkts_sent 
  • pkts_received

One can directly pick out of box available parsers within Cloud Discovery Tool. Please note that many times these parsers address all formats needs, yet one might need to add/check/correct column indices so that correct column values can be mapped and data be extracted.

Here is the list of log formats of various devices available out of the box in our current released version of Skyfence Cloud Discovery tool (Free or Enterprise). This KB article lists all supported log formats.

  • Firewalls/Web/Cloud Proxies/Routers (Native log formats exported with or without user customization)
    • Bluecoat
    • Bro
    • CEF
    • Checkpoint
    • Cisco ASA
    • Cisco ScanSafe Cloud Proxy
    • Fortigate
    • Greenplum
    • PaloAlto
    • Sonicwall
    • Squid
    • Suricata
    • Symantec Web Security Cloud
    • Websense
  • SIEM or Log Management
    • Splunk
    • McAfee Nitro
    • Syslog
  • Flow Formats (NetFlow or equivalent)
    • Cisco NetFlow

Please note that it is very easy to customize and create a new parser or .format file to support any structured log file with a delimiter.  Please read this article which elaborates the steps to customize and create a new .format file for your own log file.

How do I add a custom log format to the Skyfence Cloud Discovery Free tool?

Also refer this article to configure how to automate run of discovery scans continuously to generate daily, weekly or scheduled discovery reports which will be available for analysis in Skyfence Cloud Platform.

How to automate and schedule periodic Cloud Discovery Scans in Windows 7 (eg. Palo Alto Firewall logs)?

Have more questions? Submit a request


Powered by Zendesk