Use Python to write logs to Google Cloud

Course Notes

GENERAL COMMENTS - READ THIS FIRST!

  1. In order to best learn the concepts, watch the video first.
  2. Go through the Prerequisite Section mentioned below.
  3. Read through the technical considerations and updates.
  4. Go through the “Test Your Understanding” homework while looking at the code that comes with the course . This material covers similar concepts to the video with different code to enhance your learning experience.

PREREQUISITES

  • You need to have set up an account on Google Cloud.
  • You need to have the Cloud Logging API enabled. This means that you should see the green checkmark below:

TECHNICAL CONSIDERATIONS

LOGGING UPDATES

As of June 3, 2021, Google Cloud logging has a v2 version of the python library for logging . This library is not materially different from what is in the course video.

SECURITY CONSIDERATIONS

This code uses a credentials file to access Google Cloud. Depending on your use case, there are advanced security methods that can possibly be used to access your service account without downloading a key. Those advanced security methods are currently outside the scope of this course.


TEST YOUR UNDERSTANDING WITH CODE FROM THE COURSE

After watching the video, use the code that is supplied with the paid version of the course in order to test your understanding of logging in Python and Cloud Logging.

EXAMINE BASIC LOGGING CONFIGURATION LOCALLY AND ON THE CLOUD

Take a look at the following classes in tests/test_shared_configtools.py:

TestConfigureBasicLogging class

    
        def test_configure_basic_logging(self):
          self.assertEqual(len(logging.root.handlers), 0)
          configtools.configure_basic_logging()
          self.assertEqual(len(logging.root.handlers), 1)
    
                      
  1. Run the function test_configure_basic_logging within the TestConfigureBasicLogging class. This code is designed to assign a basic date format, assign a basic structure to the log message, and test to see if some assignment has been made to the logger. This test passes if logging.root.handlers is equal to 1? Why should this test work?
  2. 
      def test_configure_logging_local(self):
        fmt_string = '%(asctime)s.%(msecs)03d'
        fmt_string += ' | %(levelname)s'
        fmt_string += ' | %(name)s'
        fmt_string += ' | %(module)s'
        fmt_string += ' | %(funcName)s'
        fmt_string += ' | %(message)s'
    
        log_override = {'logging':
          {'console': 
            {'send': True,
             'level': 'INFO',
             'datefmt': '%Y-%m-%d %H:%M:%S',
             'format': fmt_string
            }
            , 'gcp': {'send': False}}}
    
        config = OmegaConf.merge(log_override)
        creds = None
        environ['LOG_NAME'] = 'test_logger'
        configtools.configure_logging(config.logging, creds)
    
        # This is the test of the actual logging
        log = logging.getLogger(environ['LOG_NAME'])
        log.info("This is a test")
        # log.warning("This is a warning")
        # log.error("This is an error")
    
        return
                      
  3. Use a debugger to go line by line through the test_configure_logging_local function within the TestConfigureBasicLogging class. Enable the log.warning and log.error lines in this function. What difference do you see in the console after enabling these lines?

TestLoadCredentials class

    
        def test_load_credentials(self):
          environ['CREDENTIAL_SET'] = 'USER1'
          user_cred_set = \
            configtools.load_credentials('/yaml/credentials.yml')
          
          self.assertGreaterEqual(len(user_cred_set), 1)
          self.assertNotEqual(user_cred_set['google']['project_name'], 'machinelearningmastery-123456')
    
          return
                    
  1. The test_load_credentials function in the TestLoadCredentials class confirms that you have installed the libraries for OmegaConf correctly and that you can pull a configuration file into memory. As part of a previous lesson in the Machine Learning Mastery course, you were instructed to change all occurrences of the sample project (machinelearningmastery-123456) in the yaml/credentials.yml file to the project that you have in your Google Cloud Account. Run test_load_credentials and confirm that the test works. Why should this assertNotEqual test work?

TestConfigureCloudLogging class

    
    class TestConfigureCloudLogging(unittest.TestCase):
    
      def test_configure_logging_gcp(self):
          ... 
          
          log = logging.getLogger(environ['LOG_NAME'])
          log.info("INFO - Simulated info")
          log.warning("WARNING - Simulated warning ")
          log.error("ERROR - Simulated error")
    
                        
  1. Run test_configure_logging_gcp within test_shared_configtools. Make changes to the log.info line so that you can see warnings and errors within Google Cloud Logging.

USING THE YAML FORMAT FOR CONFIGURATION FILES

Take a look at yaml/config.yml that is part of the files that are included as part of the paid course. Among other things, this file gives configuration information about logging. Also, take a look at the OmegaConf documentation for examples of a YAML implementation.

  1. OmegaConf allows the use of “dot notation” when referencing parameters in the YAML file. What string is found if config.yaml is loaded into memory as config and if config.logging.gcp.format is referenced?

Take a look at the load_configuration function in shared/configtools.py . This file is used in part to load parts of the config.yml into memory.

    
        def load_configuration(c_file, project_root):
          """
          Load full configuration data for all profiles
          found in a yaml file.
          
          :param c_file: Name of the configuration file.
          :param project_root: This is the root directory
           (e.g 'D:\\python\\machinelearningmastery')
          
           :return: Loaded file
          """
          c_file = f'{project_root}{c_file}'
          try:
              return OmegaConf.load(c_file)
          except FileNotFoundError:
              configure_basic_logging()
              logging.critical(f'Config file not found.')
              logging.critical('Exiting program.')
              raise SystemExit
          except Exception:
              configure_basic_logging()
              logging.critical(f'Error received', exc_info=True)
              logging.critical('Exiting program.')
              raise SystemExit
                    
  1. What are some reasons to have a FileNotFoundError when you are attempting to read a YAML file into memory?