User-Defined Sink: Caching logs before network interface is instantiated

[jmsargent9]

Hello,

I have implemented a User-Defined sink to write logs over a network. Here is my simplified implementation:

class NetworkSink : public sink
{
public:
    NetworkSink(INetwork& network_interface) : _network_interface(network_interface){}

    QUILL_ATTRIBUTE_HOT void write_log(/*params*/) override
    {
        _network_interface.write(log_statement);
    }

    /* ..... */

private:
    INetwork& _network_interface;
}

This works great so long as I instantiate the sink/logger after my network interface is instantiated. Unfortunately, there are a lot of things happening in my application before my network interface is instantiated, and I would really like to be able to use the logger immediately at application startup. Therefore, if possible, I'd like to cache logs that happen before my network interface is instantiated and then send them as soon as the network interface is instantiated.

Is what I am describing possible? Or is there another approach that I could take?

Thank you!

[odygrd]

hey, would something like this work ? You will need to log something on that Sink after the network interface is instantiated in order to trigger clearing the cache.
It is also possible to use run_periodic_tasks() for doing this if you log nothing after the network interface is instantiated but it's simpler to just log something because you don't know if run_periodic_tasks() or write_log() will be called next, you will need an additional variable if run_periodic_tasks() is used to clear the cache after the network instantiaton

class UserSink final : public quill::Sink
{
public:
  UserSink() = default;

  /***/
  void write_log(quill::MacroMetadata const* /** log_metadata **/, uint64_t /** log_timestamp **/,
                 std::string_view /** thread_id **/, std::string_view /** thread_name **/,
                 std::string const& /** process_id **/, std::string_view /** logger_name **/,
                 quill::LogLevel /** log_level **/, std::string_view /** log_level_description **/,
                 std::string_view /** log_level_short_code **/,
                 std::vector<std::pair<std::string, std::string>> const* /** named_args - only populated when named args in the format placeholder are used **/,
                 std::string_view /** log_message **/, std::string_view log_statement) override
  {
    if (!_network_interface.load(std::memory_order_acquire))
    {
      _cached_log_statements.push_back(std::string{log_statement.data(), log_statement.size() - 1});
      return;
    }

    if (!_cached_log_statements.empty())
    {
      for (auto const& elem : _cached_log_statements)
      {
        _network_interface->write(elem);
      }

      _cached_log_statements.clear();
    }

    _network_interface->write(log_statement);
  }

  /***/
  void flush_sink() noexcept override
  {
  }

  /***/
  void run_periodic_tasks() noexcept override
  {
  }

  void set_network_interface(INetwork* network_interface)
  {
    // call this when the network interface is instantiated
    _network_interface.store(network_interface, std::memory_order_release);

   // do not clear the cache in this function as is called from another thread, let the backend worker thread do it instead in 
   // write_log()
  }

private:
  std::vector<std::string> _cached_log_statements;
  std::atomic<INetwork*> _network_interface {nullptr};
};

you can get your sink after creation like this

creation

  auto sink = quill::Frontend::create_or_get_sink<UserSink>("your-unique-name-or-id");
  quill::Logger* logger = quill::Frontend::create_or_get_logger("root", std::move(sink));

when Network Interface is instantiated

  auto network_sink = reinterpret_cast<UserSink*>(quill::Frontend::get_sink("your-unique-name-or-id").get());
  network_sink->set_network_interface(...);

[jmsargent9]

This is the type of approach I was looking for, as it is more compatible with my implementation. I can't thank you enough for your assistance with this!

[odygrd]

another solution, probably cleaner one, if you add a flag to the INetwork

class INetwork
{
public:
  virtual void write(std::string_view s) {};

  bool is_ready() const noexcept
  {
    return _is_ready.load(std::memory_order_acquire);
  }

protected:
  void set_is_ready(bool value) noexcept
  {
    _is_ready.store(value, std::memory_order_release);
  }

private:
  std::atomic<bool> _is_ready {false};
};

struct UDP : public INetwork
{
  void init()
  {
    // ...

    set_is_ready(true);
  }

};

class UserSink final : public quill::Sink
{
public:
  UserSink(INetwork& network_interface) : _network_interface(network_interface) {}

  /***/
  void write_log(quill::MacroMetadata const* /** log_metadata **/, uint64_t /** log_timestamp **/,
                 std::string_view /** thread_id **/, std::string_view /** thread_name **/,
                 std::string const& /** process_id **/, std::string_view /** logger_name **/,
                 quill::LogLevel /** log_level **/, std::string_view /** log_level_description **/,
                 std::string_view /** log_level_short_code **/,
                 std::vector<std::pair<std::string, std::string>> const* /** named_args - only populated when named args in the format placeholder are used **/,
                 std::string_view /** log_message **/, std::string_view log_statement) override
  {
    if (!_network_interface.is_ready())
    {
      _cached_log_statements.push_back(std::string{log_statement.data(), log_statement.size() - 1});
      return;
    }

    if (!_cached_log_statements.empty())
    {
      for (auto const& elem : _cached_log_statements)
      {
        _network_interface.write(elem);
      }

      _cached_log_statements.clear();
    }

    _network_interface.write(log_statement);
  }

  /***/
  void flush_sink() noexcept override
  {
  }

  /***/
  void run_periodic_tasks() noexcept override
  {
  }

private:
  INetwork& _network_interface;
  std::vector<std::string> _cached_log_statements;
};