-->

You can configure automatic log upload for continuous reports in Cloud App Security using a Docker on Ubuntu, Red Hat Enterprise Linux (RHEL), or CentOS in Azure.

Prerequisites

• OS:

  • Ubuntu 14.04, 16.04, and 18.04
  • RHEL 7.2 or higher
  • CentOS 7.2 or higher

• Disk space: 250 GB

• CPU: 2

• RAM: 4 GB

• Set your firewall as described in Network requirements

Hi Mario, Yep I checked the BIOS, ensured virtualization was enabled and secure boot disabled. I also checked within Windows (Task Manager - Performance tab), and checked that Windows was displaying ‘Virtualization - enabled’.

Docker CE Stable - x8664 14 kB/s 3.8 kB 00:00 Red Hat Enterprise Linux 8 for x8664 - AppStream (RPMs) 33 kB/s 2.8 kB 00:00 Red Hat Ansible Engine 2.8 for RHEL 8 x8664 (RPMs) 29 kB/s 2.4 kB 00:00 Red Hat Enterprise Linux 8 for x8664 - BaseOS (RPMs) 30 kB/s 2.4 kB 00:00 Red Hat Satellite Tools 6.6 for RHEL 8 x8664 (RPMs) 29 kB/s 2. Done The following additional packages will be installed: aufs-tools cgroupfs-mount pigz The following NEW packages will be installed: aufs-tools cgroupfs-mount containerd.io docker-ce docker-ce-cli pigz 0 upgraded, 6 newly installed, 0 to remove and 167 not upgraded. Need to get 50.7 MB of archives.

Note

If you have an existing log collector and want to remove it before deploying it again, or if you simply want to remove it, run the following commands:

Log collector performance

The Log collector can successfully handle log capacity of up to 50 GB per hour comprised of up to 10 data sources. The main bottlenecks in the log collection process are:

• With everything set, you can finally move on to installing Docker on CentOS 7 by running: sudo yum install docker. The system should begin the installation. Once it finishes, it will notify you the installation is complete and which version of Docker is now running on your system.
• In this article. You can configure automatic log upload for continuous reports in Cloud App Security using a Docker on an on-premises Ubuntu, Red Hat Enterprise Linux (RHEL), or CentOS server.

• Network bandwidth - Your network bandwidth determines the log upload speed.

• I/O performance of the virtual machine - Determines the speed at which logs are written to the log collector's disk. The log collector has a built-in safety mechanism that monitors the rate at which logs arrive and compares it to the upload rate. In cases of congestion, the log collector starts to drop log files. If your setup typically exceeds 50 GB per hour, we recommend that you split the traffic between multiple log collectors.

Note

If you require more than 10 data sources, we recommend that you split the data sources between multiple log collectors.

Set up and configuration

Step 1 – Web portal configuration: Define data sources and link them to a log collector

1. Go to the Automatic log upload settings page.

   1. In the Cloud App Security portal, click the settings icon followed by Log collectors.

2. For each firewall or proxy from which you want to upload logs, create a matching data source.

   1. Click Add data source.
      
   2. Name your proxy or firewall.
      
   3. Select the appliance from the Source list. If you select Custom log format to work with a network appliance that isn't listed, see Working with the custom log parser for configuration instructions.
   4. Compare your log with the sample of the expected log format. If your log file format doesn't match this sample, you should add your data source as Other.
   5. Set the Receiver type to either FTP, FTPS, Syslog – UDP, or Syslog – TCP, or Syslog – TLS.

   Note

   Integrating with secure transfer protocols (FTPS and Syslog – TLS) often requires additional settings or your firewall/proxy.

   f. Repeat this process for each firewall and proxy whose logs can be used to detect traffic on your network. It's recommended to set up a dedicated data source per network device to enable you to:

   • Monitor the status of each device separately, for investigation purposes.
   • Explore Shadow IT Discovery per device, if each device is used by a different user segment.

3. Go to the Log collectors tab at the top.

   1. Click Add log collector.
   2. Give the log collector a name.
   3. Enter the Host IP address of the machine you'll use to deploy the Docker. The host IP address can be replaced with the machine name, if there is a DNS server (or equivalent) that will resolve the host name.
   4. Select all Data sources that you want to connect to the collector, and click Update to save the configuration.
      

4. Further deployment information will appear. Copy the run command from the dialog. You can use the copy to clipboard icon.

5. Export the expected data source configuration. This configuration describes how you should set the log export in your appliances.

   Note

   • A single Log collector can handle multiple data sources.
   • Copy the contents of the screen because you will need the information when you configure the Log Collector to communicate with Cloud App Security. If you selected Syslog, this information will include information about which port the Syslog listener is listening on.
   • For users sending log data via FTP for the first time, we recommend changing the password for the FTP user. For more information, see Changing the FTP password.

Step 2 – Deployment of your machine in Azure

Note

The following steps describe the deployment in Ubuntu. The deployment steps for other platforms are slightly different.

1. Create a new Ubuntu machine in your Azure environment.

2. After the machine is up, open the ports by:

   1. In the machine view, go to Networking select the relevant interface by double-clicking on it.
   2. Go to Network security group and select the relevant network security group.
   3. Go to Inbound security rules and click Add,
   4. Add the following rules (in Advanced mode):

   Name	Destination port ranges	Protocol	Source	Destination
   caslogcollector_ftp	21	TCP	<Your appliance's IP address's subnet>	Any
   caslogcollector_ftp_passive	20000-20099	TCP	<Your appliance's IP address's subnet>	Any
   caslogcollector_syslogs_tcp	601-700	TCP	<Your appliance's IP address's subnet>	Any
   caslogcollector_syslogs_udp	514-600	UDP	<Your appliance's IP address's subnet>	Any

3. Go back to the machine and click Connect to open a terminal on the machine.

4. Change to root privileges using sudo -i.

5. If you accept the software license terms, uninstall old versions and install Docker CE by running the commands appropriate for your environment:

1. Remove old versions of Docker: yum erase docker docker-engine docker.io

2. Install Docker engine prerequisites: yum install -y yum-utils

3. Add Docker repository:

4. Install Docker engine: yum -y install docker-ce

5. Start Docker

6. Test Docker installation: docker run hello-world

1. Remove old versions of Docker: yum erase docker docker-engine docker.io

2. Install Docker engine prerequisites:

3. Add Docker repository:

4. Install dependencies:

5. Install Docker engine: sudo yum install docker-ce

6. Start Docker

7. Test Docker installation: docker run hello-world

1. Remove the container-tools module: yum module remove container-tools

2. Add the Docker CE repository: yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

3. Modify the yum repo file to use CentOS 8/RHEL 8 packages: sed -i s/7/8/g /etc/yum.repos.d/docker-ce.repo

4. Install Docker CE: yum install docker-ce

5. Start Docker

6. Test Docker installation: docker run hello-world

1. Remove old versions of Docker: apt-get remove docker docker-engine docker.io

2. If you are installing on Ubuntu 14.04, install the linux-image-extra package.

3. Install Docker engine prerequisites:

4. Verify that the apt-key fingerprint UID is docker@docker.com: apt-key fingerprint | grep uid

5. Install Docker engine:

6. Test Docker installation: docker run hello-world

1. In the Cloud App Security portal in the Create new log collector window, copy the command to import the collector configuration on the hosting machine:

2. Run the command to deploy the log collector.

3. To verify that the log collector is running properly, run the following command: Docker logs <collector_name>. You should get the results: Finished successfully!

Step 3 - On-premises configuration of your network appliances

Configure your network firewalls and proxies to periodically export logs to the dedicated Syslog port of the FTP directory according to the directions in the dialog. For example:

Step 4 - Verify the successful deployment in the Cloud App Security portal

Check the collector status in the Log collector table and make sure the status is Connected. If it's Created, it's possible the log collector connection and parsing haven't completed.

Install Docker Ce On Rhel 7.8

You can also go to the Governance log and verify that logs are being periodically uploaded to the portal.

Alternatively, you can check the log collector status from within the docker container using the following commands:

1. Log in to the container by using this command: docker exec -it <Container Name> bash
2. Verify the log collector status using this command: collector_status -p

If you have problems during deployment, see Troubleshooting Cloud Discovery.

Optional - Create custom continuous reports

Verify that the logs are being uploaded to Cloud App Security and that reports are generated. After verification, create custom reports. You can create custom discovery reports based on Azure Active Directory user groups. For example, if you want to see the cloud use of your marketing department, import the marketing group using the import user group feature. Then create a custom report for this group. You can also customize a report based on IP address tag or IP address ranges.

1. In the Cloud App Security portal, under the Settings cog, select Cloud Discovery settings, and then select Continuous reports.

2. Click the Create report button and fill in the fields.

3. Under the Filters you can filter the data by data source, by imported user group, or by IP address tags and ranges.

Next steps

If you run into any problems, we're here to help. To get assistance or support for your product issue, please open a support ticket.

-->

You can configure automatic log upload for continuous reports in Cloud App Security using a Docker on an on-premises Ubuntu, Red Hat Enterprise Linux (RHEL), or CentOS server.

Prerequisites

• OS:

  • Ubuntu 14.04, 16.04, and 18.04
  • RHEL 7.2 or higher
  • CentOS 7.2 or higher

• Disk space: 250 GB

• CPU: 2

• RAM: 4 GB

• Set your firewall as described in Network requirements

Note

If you have an existing log collector and want to remove it before deploying it again, or if you simply want to remove it, run the following commands:

Log collector performance

The Log collector can successfully handle log capacity of up to 50 GB per hour. The main bottlenecks in the log collection process are:

• Network bandwidth - Your network bandwidth determines the log upload speed.

• I/O performance of the virtual machine - Determines the speed at which logs are written to the log collector's disk. The log collector has a built-in safety mechanism that monitors the rate at which logs arrive and compares it to the upload rate. In cases of congestion, the log collector starts to drop log files. If your setup typically exceeds 50 GB per hour, it's recommended that you split the traffic between multiple log collectors.

Set up and configuration

Step 1 – Web portal configuration: Define data sources and link them to a log collector

Install Docker Ce On Rhel 7.7

1. Go to the Automatic log upload settings page.

   1. In the Cloud App Security portal, click the settings icon followed by Log collectors.

2. For each firewall or proxy from which you want to upload logs, create a matching data source.

   1. Click Add data source.
      
   2. Name your proxy or firewall.
      
   3. Select the appliance from the Source list. If you select Custom log format to work with a network appliance that isn't listed, see Working with the custom log parser for configuration instructions.
   4. Compare your log with the sample of the expected log format. If your log file format doesn't match this sample, you should add your data source as Other.
   5. Set the Receiver type to either FTP, FTPS, Syslog – UDP, or Syslog – TCP, or Syslog – TLS.

   Note

   Integrating with secure transfer protocols (FTPS and Syslog – TLS) often requires additional settings or your firewall/proxy.

   f. Repeat this process for each firewall and proxy whose logs can be used to detect traffic on your network. It's recommended to set up a dedicated data source per network device to enable you to:

   • Monitor the status of each device separately, for investigation purposes.
   • Explore Shadow IT Discovery per device, if each device is used by a different user segment.

3. Go to the Log collectors tab at the top.

   1. Click Add log collector.
   2. Give the log collector a name.
   3. Enter the Host IP address of the machine you'll use to deploy the Docker. The host IP address can be replaced with the machine name, if there is a DNS server (or equivalent) that will resolve the host name.
   4. Select all Data sources that you want to connect to the collector, and click Update to save the configuration.

4. Further deployment information will appear. Copy the run command from the dialog. You can use the copy to clipboard icon.

5. Export the expected data source configuration. This configuration describes how you should set the log export in your appliances.

   Note

   • A single Log collector can handle multiple data sources.
   • Copy the contents of the screen because you will need the information when you configure the Log Collector to communicate with Cloud App Security. If you selected Syslog, this information will include information about which port the Syslog listener is listening on.
   • For users sending log data via FTP for the first time, we recommend changing the password for the FTP user. For more information, see Changing the FTP password.

Step 2 – On-premises deployment of your machine

Upgrade Docker Ce On Rhel 7

The following steps describe the deployment in Ubuntu.

Note

The deployment steps for other supported platforms may be slightly different.

1. Open a terminal on your Ubuntu machine.

2. Change to root privileges using the command: sudo -i

3. To bypass a proxy in your network, run the following two commands:

4. If you accept the software license terms, uninstall old versions and install Docker CE by running the commands appropriate for your environment:

1. Remove old versions of Docker: yum erase docker docker-engine docker.io

2. Install Docker engine prerequisites: yum install -y yum-utils

3. Add Docker repository:

4. Install Docker engine: yum -y install docker-ce

5. Start Docker

6. Test Docker installation: docker run hello-world

1. Remove old versions of Docker: yum erase docker docker-engine docker.io

2. Install Docker engine prerequisites:

3. Add Docker repository:

4. Install dependencies:

5. Install Docker engine: sudo yum install docker-ce

6. Start Docker

7. Test Docker installation: docker run hello-world

1. Remove the container-tools module: yum module remove container-tools

2. Add the Docker CE repository: yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

3. Modify the yum repo file to use CentOS 8/RHEL 8 packages: sed -i s/7/8/g /etc/yum.repos.d/docker-ce.repo

4. Install Docker CE: yum install docker-ce

5. Start Docker

6. Test Docker installation: docker run hello-world

1. Remove old versions of Docker: apt-get remove docker docker-engine docker.io

2. If you are installing on Ubuntu 14.04, install the linux-image-extra package.

3. Install Docker engine prerequisites:

4. Verify that the apt-key fingerprint UID is docker@docker.com: apt-key fingerprint | grep uid

5. Install Docker engine:

6. Test Docker installation: docker run hello-world

5. Deploy the collector image on the hosting machine by importing the collector configuration. Import the configuration by copying the run command generated in the portal. If you need to configure a proxy, add the proxy IP address and port number. For example, if your proxy details are 192.168.10.1:8080, your updated run command is:

6. Verify that the collector is running properly with the following command: docker logs <collector_name>

You should see the message: Finished successfully!

Step 3 - On-premises configuration of your network appliances

Configure your network firewalls and proxies to periodically export logs to the dedicated Syslog port or the FTP directory according to the directions in the dialog. For example:

Step 4 - Verify the successful deployment in the Cloud App Security portal

Check the collector status in the Log collector table and make sure the status is Connected. If it's Created, it's possible the log collector connection and parsing haven't completed.

You can also go to the Governance log and verify that logs are being periodically uploaded to the portal.

Alternatively, you can check the log collector status from within the docker container using the following commands:

1. Log in to the container by using this command: docker exec -it <Container Name> bash
2. Verify the log collector status using this command: collector_status -p

If you have problems during deployment, see Troubleshooting Cloud Discovery.

Optional - Create custom continuous reports

Verify that the logs are being uploaded to Cloud App Security and that reports are generated. After verification, create custom reports. You can create custom discovery reports based on Azure Active Directory user groups. For example, if you want to see the cloud use of your marketing department, import the marketing group using the import user group feature. Then create a custom report for this group. You can also customize a report based on IP address tag or IP address ranges.

1. In the Cloud App Security portal, under the Settings cog, select Cloud Discovery settings, and then select Continuous reports.
2. Click the Create report button and fill in the fields.
3. Under the Filters you can filter the data by data source, by imported user group, or by IP address tags and ranges.

Next steps

Docker Ce On Rhel 7.6

If you run into any problems, we're here to help. To get assistance or support for your product issue, please open a support ticket.