Multiple burst filters

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Multiple burst filters

bharat naik
Hi,

I am using log4j2 in my project (2.11.1) and there are several places in my
code where I need to throttle the logging, in case of exceptions, and hence
wanted to use the burst filter.

However, I don’t see a way of using multiple burst filters with different
rate and maxBurst for different logging invocations, all going to the same
log file.

For example, what I have so far is:

<Filters>
        <MarkerFilter marker="THROTTLE"onMatch="NEUTRAL" onMismatch="ACCEPT"
/>
        <BurstFilter level="WARN" rate="1"maxBurst="1”/>
</Filters>

and invoking it via:

A()
{
..
logger.warn(throttle, “msg”); <—throttle is a marker for “THROTTLE"
..
}

B()
{
..
logger.warn(throttle, “msg”);
..
}

However, instead if I could have multiple burst filters for different
markers that would be ideal since currently I could miss log messages from
B() and only log A() since they all go through one burst filter. Basically,
I want to ensure at least one log output for every call to logger.warn(..),
and throttle each of them independently.

Is that possible?

Thanks,
Bharat.
Reply | Threaded
Open this post in threaded view
|

Re: Multiple burst filters

Ralph Goers
Your filter configuration says “If this is an event with a throttle marker continue checking. If it is not a throttle marker log it, regardless of its logging level.” Is that really what you want?

If you want to throttle different calls to different loggers independently than just put the burst filter on the logger or the logger’s appender reference.

Ralph

> On Jul 27, 2020, at 10:32 PM, bharat naik <[hidden email]> wrote:
>
> Hi,
>
> I am using log4j2 in my project (2.11.1) and there are several places in my
> code where I need to throttle the logging, in case of exceptions, and hence
> wanted to use the burst filter.
>
> However, I don’t see a way of using multiple burst filters with different
> rate and maxBurst for different logging invocations, all going to the same
> log file.
>
> For example, what I have so far is:
>
> <Filters>
>        <MarkerFilter marker="THROTTLE"onMatch="NEUTRAL" onMismatch="ACCEPT"
> />
>        <BurstFilter level="WARN" rate="1"maxBurst="1”/>
> </Filters>
>
> and invoking it via:
>
> A()
> {
> ..
> logger.warn(throttle, “msg”); <—throttle is a marker for “THROTTLE"
> ..
> }
>
> B()
> {
> ..
> logger.warn(throttle, “msg”);
> ..
> }
>
> However, instead if I could have multiple burst filters for different
> markers that would be ideal since currently I could miss log messages from
> B() and only log A() since they all go through one burst filter. Basically,
> I want to ensure at least one log output for every call to logger.warn(..),
> and throttle each of them independently.
>
> Is that possible?
>
> Thanks,
> Bharat.



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Multiple burst filters

bharat naik
Hi Ralph,

Yes, that is the behavior I want ie all logs without a marker need to get
printed because they are placed in such a way that they don’t need any
throttling. The ones that need throttling will be logged using the marker.

However, I want different throttling for different markers. They all go to
the same log ie there is only one logger/appender.

So, is it possible to assign burst filters at a marker level?

Thanks,
Bharat.

On Mon, Jul 27, 2020 at 11:03 PM Ralph Goers <[hidden email]>
wrote:

> Your filter configuration says “If this is an event with a throttle marker
> continue checking. If it is not a throttle marker log it, regardless of its
> logging level.” Is that really what you want?
>
> If you want to throttle different calls to different loggers independently
> than just put the burst filter on the logger or the logger’s appender
> reference.
>
> Ralph
>
> > On Jul 27, 2020, at 10:32 PM, bharat naik <[hidden email]> wrote:
> >
> > Hi,
> >
> > I am using log4j2 in my project (2.11.1) and there are several places in
> my
> > code where I need to throttle the logging, in case of exceptions, and
> hence
> > wanted to use the burst filter.
> >
> > However, I don’t see a way of using multiple burst filters with different
> > rate and maxBurst for different logging invocations, all going to the
> same
> > log file.
> >
> > For example, what I have so far is:
> >
> > <Filters>
> >        <MarkerFilter marker="THROTTLE"onMatch="NEUTRAL"
> onMismatch="ACCEPT"
> > />
> >        <BurstFilter level="WARN" rate="1"maxBurst="1”/>
> > </Filters>
> >
> > and invoking it via:
> >
> > A()
> > {
> > ..
> > logger.warn(throttle, “msg”); <—throttle is a marker for “THROTTLE"
> > ..
> > }
> >
> > B()
> > {
> > ..
> > logger.warn(throttle, “msg”);
> > ..
> > }
> >
> > However, instead if I could have multiple burst filters for different
> > markers that would be ideal since currently I could miss log messages
> from
> > B() and only log A() since they all go through one burst filter.
> Basically,
> > I want to ensure at least one log output for every call to
> logger.warn(..),
> > and throttle each of them independently.
> >
> > Is that possible?
> >
> > Thanks,
> > Bharat.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Multiple burst filters

Ralph Goers
Just to make sure we are clear - ALL messages that don’t have a THROTTLE marker will be logged - trace, debug, etc from every component., including any third party libraries you are using. That would be highly unusual.

It might help if you provided your complete logging configuration and a complete sample class.  It isn’t clear to me why you will only have one Logger in your application. Surely other classes and third party libraries you might be using will refer to others.

I also don’t understand what you mean by different throttling by different markers. Here you have only shown a single Marker.

If you really only have a single Logger (which I find very hard to believe) but somehow have multiple markers then you are going to have a hard time with this.

Ralph

> On Jul 28, 2020, at 3:31 PM, bharat naik <[hidden email]> wrote:
>
> Hi Ralph,
>
> Yes, that is the behavior I want ie all logs without a marker need to get
> printed because they are placed in such a way that they don’t need any
> throttling. The ones that need throttling will be logged using the marker.
>
> However, I want different throttling for different markers. They all go to
> the same log ie there is only one logger/appender.
>
> So, is it possible to assign burst filters at a marker level?
>
> Thanks,
> Bharat.
>
> On Mon, Jul 27, 2020 at 11:03 PM Ralph Goers <[hidden email]>
> wrote:
>
>> Your filter configuration says “If this is an event with a throttle marker
>> continue checking. If it is not a throttle marker log it, regardless of its
>> logging level.” Is that really what you want?
>>
>> If you want to throttle different calls to different loggers independently
>> than just put the burst filter on the logger or the logger’s appender
>> reference.
>>
>> Ralph
>>
>>> On Jul 27, 2020, at 10:32 PM, bharat naik <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> I am using log4j2 in my project (2.11.1) and there are several places in
>> my
>>> code where I need to throttle the logging, in case of exceptions, and
>> hence
>>> wanted to use the burst filter.
>>>
>>> However, I don’t see a way of using multiple burst filters with different
>>> rate and maxBurst for different logging invocations, all going to the
>> same
>>> log file.
>>>
>>> For example, what I have so far is:
>>>
>>> <Filters>
>>>       <MarkerFilter marker="THROTTLE"onMatch="NEUTRAL"
>> onMismatch="ACCEPT"
>>> />
>>>       <BurstFilter level="WARN" rate="1"maxBurst="1”/>
>>> </Filters>
>>>
>>> and invoking it via:
>>>
>>> A()
>>> {
>>> ..
>>> logger.warn(throttle, “msg”); <—throttle is a marker for “THROTTLE"
>>> ..
>>> }
>>>
>>> B()
>>> {
>>> ..
>>> logger.warn(throttle, “msg”);
>>> ..
>>> }
>>>
>>> However, instead if I could have multiple burst filters for different
>>> markers that would be ideal since currently I could miss log messages
>> from
>>> B() and only log A() since they all go through one burst filter.
>> Basically,
>>> I want to ensure at least one log output for every call to
>> logger.warn(..),
>>> and throttle each of them independently.
>>>
>>> Is that possible?
>>>
>>> Thanks,
>>> Bharat.
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Multiple burst filters

bharat naik
This is my log4j2.xml file:

<?xml version="1.0" encoding="UTF-8" ?>

<Configuration>

  <Filters>

    <MarkerFilter marker="THROTTLE" onMatch="NEUTRAL" onMismatch="ACCEPT"/>

    <BurstFilter level="WARN" rate="1" maxBurst="1"/>

  </Filters>

  <Appenders>


    <RollingFile name="R" fileName="/var/log/project/logs/data/output.log"
bufferedIO="true" immediateFlush="false"


filePattern="/var/log/project/logs/project/output.log.%d{yyyy-MM-dd-HH}">

      <ThresholdFilter level="INFO" onMatch="ACCEPT" onMismatch="DENY"/>
            <--- I only use info, warn and error in my logging

      <PatternLayout pattern="%d{MM/dd/yy HH:mm:ss,SSS} [%t] %p [%c]
(%F:%L) - %m%n"/>

      <TimeBasedTriggeringPolicy modulate="true"/>

    </RollingFile>

  </Appenders>



  <Loggers>

    <Logger name="org.project.clips" level="INFO">

      <AppenderRef ref="clipslog" />

    </Logger>

    <Root level="INFO">

      <AppenderRef ref="R"/>

    </Root>

  </Loggers>


</Configuration>


The Throttle marker is used for messages that I want to be throttled
because when the particular condition hits, it hits repeatedly and floods
my logs. So I want to throttle it.


Other places where logging occurs, I want them to be left as-is. If they
need throttling, I'll add the marker there too. Currently, I have narrowed
down about 10 places in my code base that require throttling of logs.


I have shown a single marker because currently I am not able to figure out
how to throttle differently based on different markers.


What I ideally want is something like:


<MarkerFilter marker="THROTTLE_LOW"> -> burst filter with rate=10, maxBurst
= 100

<MarkerFilter marker="THROTTLE_MED"> -> burst filter with rate = 5,
maxBurst = 10

<MarkerFilter marker="THROTTLE_HI"> -> burst filter with rate = 1, maxBurst
= 1


So, basically different burst filters associated with different markers.


If this is possible, I would use them as:


public class A

{

   private static Logger logger = LoggerFactory.getLogger(A.class);

   A()

   {

     ..

     catch(Exception ex)

     {

       logger.warn(throttle_low, "msg1");

     }

   }

}//class A ends


public class B

{

    private static Logger logger = LoggerFactory.getLogger(B.class);

    B()

    {

      ..

      catch(Exceptions ex)

      {

        logger.warn(throttle_hi, "msg2");

      }

    }

}// class B ends



The problem I have with using the same marker, say throttle_hi, everywhere
is that there is no guarantee that msg2 will get printed (i.e. msg1 might
always get printed) because it occurred earlier.


Thanks,

Bharat.



On Tue, Jul 28, 2020 at 4:01 PM Ralph Goers <[hidden email]>
wrote:

> Just to make sure we are clear - ALL messages that don’t have a THROTTLE
> marker will be logged - trace, debug, etc from every component., including
> any third party libraries you are using. That would be highly unusual.
>
> It might help if you provided your complete logging configuration and a
> complete sample class.  It isn’t clear to me why you will only have one
> Logger in your application. Surely other classes and third party libraries
> you might be using will refer to others.
>
> I also don’t understand what you mean by different throttling by different
> markers. Here you have only shown a single Marker.
>
> If you really only have a single Logger (which I find very hard to
> believe) but somehow have multiple markers then you are going to have a
> hard time with this.
>
> Ralph
>
> > On Jul 28, 2020, at 3:31 PM, bharat naik <[hidden email]> wrote:
> >
> > Hi Ralph,
> >
> > Yes, that is the behavior I want ie all logs without a marker need to get
> > printed because they are placed in such a way that they don’t need any
> > throttling. The ones that need throttling will be logged using the
> marker.
> >
> > However, I want different throttling for different markers. They all go
> to
> > the same log ie there is only one logger/appender.
> >
> > So, is it possible to assign burst filters at a marker level?
> >
> > Thanks,
> > Bharat.
> >
> > On Mon, Jul 27, 2020 at 11:03 PM Ralph Goers <[hidden email]
> >
> > wrote:
> >
> >> Your filter configuration says “If this is an event with a throttle
> marker
> >> continue checking. If it is not a throttle marker log it, regardless of
> its
> >> logging level.” Is that really what you want?
> >>
> >> If you want to throttle different calls to different loggers
> independently
> >> than just put the burst filter on the logger or the logger’s appender
> >> reference.
> >>
> >> Ralph
> >>
> >>> On Jul 27, 2020, at 10:32 PM, bharat naik <[hidden email]>
> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am using log4j2 in my project (2.11.1) and there are several places
> in
> >> my
> >>> code where I need to throttle the logging, in case of exceptions, and
> >> hence
> >>> wanted to use the burst filter.
> >>>
> >>> However, I don’t see a way of using multiple burst filters with
> different
> >>> rate and maxBurst for different logging invocations, all going to the
> >> same
> >>> log file.
> >>>
> >>> For example, what I have so far is:
> >>>
> >>> <Filters>
> >>>       <MarkerFilter marker="THROTTLE"onMatch="NEUTRAL"
> >> onMismatch="ACCEPT"
> >>> />
> >>>       <BurstFilter level="WARN" rate="1"maxBurst="1”/>
> >>> </Filters>
> >>>
> >>> and invoking it via:
> >>>
> >>> A()
> >>> {
> >>> ..
> >>> logger.warn(throttle, “msg”); <—throttle is a marker for “THROTTLE"
> >>> ..
> >>> }
> >>>
> >>> B()
> >>> {
> >>> ..
> >>> logger.warn(throttle, “msg”);
> >>> ..
> >>> }
> >>>
> >>> However, instead if I could have multiple burst filters for different
> >>> markers that would be ideal since currently I could miss log messages
> >> from
> >>> B() and only log A() since they all go through one burst filter.
> >> Basically,
> >>> I want to ensure at least one log output for every call to
> >> logger.warn(..),
> >>> and throttle each of them independently.
> >>>
> >>> Is that possible?
> >>>
> >>> Thanks,
> >>> Bharat.
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: [hidden email]
> >> For additional commands, e-mail: [hidden email]
> >>
> >>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>