JsonLayout Configuration

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

JsonLayout Configuration

Ayush Kumar
Not sure where else to ask this question (stackoverflow maybe?), so
apologies if this isn't the right forum.

We've recently migrated to log4j2 from logback and are using the jsonlayout
for a file appender. These logs are then parsed by scalyr (awesome tool
btw). We've noticed that extended stacktraces in jsonlayout are
ridiculously long. Reaching >20k characters, even with string mode enabled.
This is a problem because there's a max log line size that we can parse
(something around 8k I think).

I was wondering if there was any way to get the default jsonlayout to cut
down the size of stacktraces. PatternLayout has %ex{n}, and I was hoping
for something that wasn't creating my own layout, rewriting every log
message, or manually making pattern layout be a json format with %enc{json}
and a pattern like {"message": "%enc(%m){json}", ...}.

I've created a ticket for it as well
https://issues.apache.org/jira/browse/LOG4J2-2879.

Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: JsonLayout Configuration

Volkan Yazıcı
Hello Ayush,

As of this moment, there doesn't exist such a configuration knob in
JsonLayout, AFAIK. Though the future is bright! There is an external
plugin, log4j2-logstash-layout[1], which delivers what you're looking
for (see its maxStringLength parameter) and more! The good news is, we
are in the process of merging this external plugin into the Log4j core
with a major overhaul and rebranding, JsonTemplateLayout.
Implementation is completed, but it is waiting for a final review on a
particular aspect by one of the core members, Ralph Goers.

My personal advice would be to go with LogstashLayout and switch to
JsonTemplateLayout once it is there.

Kind regards.

[1] https://github.com/vy/log4j2-logstash-layout

On Tue, Jun 30, 2020 at 8:47 PM Ayush Kumar <[hidden email]> wrote:

>
> Not sure where else to ask this question (stackoverflow maybe?), so
> apologies if this isn't the right forum.
>
> We've recently migrated to log4j2 from logback and are using the jsonlayout
> for a file appender. These logs are then parsed by scalyr (awesome tool
> btw). We've noticed that extended stacktraces in jsonlayout are
> ridiculously long. Reaching >20k characters, even with string mode enabled.
> This is a problem because there's a max log line size that we can parse
> (something around 8k I think).
>
> I was wondering if there was any way to get the default jsonlayout to cut
> down the size of stacktraces. PatternLayout has %ex{n}, and I was hoping
> for something that wasn't creating my own layout, rewriting every log
> message, or manually making pattern layout be a json format with %enc{json}
> and a pattern like {"message": "%enc(%m){json}", ...}.
>
> I've created a ticket for it as well
> https://issues.apache.org/jira/browse/LOG4J2-2879.
>
> Thanks.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: JsonLayout Configuration

Ralph Goers
If you can live with the GelfLayout you could also try that. It will let you format the message attribute using a Pattern String equivalent to the PatternLayout, so you use its expression to limit the length.

Ralph

> On Jun 30, 2020, at 12:28 PM, Volkan Yazıcı <[hidden email]> wrote:
>
> Hello Ayush,
>
> As of this moment, there doesn't exist such a configuration knob in
> JsonLayout, AFAIK. Though the future is bright! There is an external
> plugin, log4j2-logstash-layout[1], which delivers what you're looking
> for (see its maxStringLength parameter) and more! The good news is, we
> are in the process of merging this external plugin into the Log4j core
> with a major overhaul and rebranding, JsonTemplateLayout.
> Implementation is completed, but it is waiting for a final review on a
> particular aspect by one of the core members, Ralph Goers.
>
> My personal advice would be to go with LogstashLayout and switch to
> JsonTemplateLayout once it is there.
>
> Kind regards.
>
> [1] https://github.com/vy/log4j2-logstash-layout
>
> On Tue, Jun 30, 2020 at 8:47 PM Ayush Kumar <[hidden email]> wrote:
>>
>> Not sure where else to ask this question (stackoverflow maybe?), so
>> apologies if this isn't the right forum.
>>
>> We've recently migrated to log4j2 from logback and are using the jsonlayout
>> for a file appender. These logs are then parsed by scalyr (awesome tool
>> btw). We've noticed that extended stacktraces in jsonlayout are
>> ridiculously long. Reaching >20k characters, even with string mode enabled.
>> This is a problem because there's a max log line size that we can parse
>> (something around 8k I think).
>>
>> I was wondering if there was any way to get the default jsonlayout to cut
>> down the size of stacktraces. PatternLayout has %ex{n}, and I was hoping
>> for something that wasn't creating my own layout, rewriting every log
>> message, or manually making pattern layout be a json format with %enc{json}
>> and a pattern like {"message": "%enc(%m){json}", ...}.
>>
>> I've created a ticket for it as well
>> https://issues.apache.org/jira/browse/LOG4J2-2879.
>>
>> Thanks.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: JsonLayout Configuration

Ayush Kumar
Thank you Volkan and Ralph for your ridiculously fast responses. I think we
can put in the, what seems to be pretty minimal, effort to get the logstash
layout plugin working and then migrate to JsonTemplateLayout when that
releases. The ability to order fields, while less critical, is also a nice
boon as well.

On Tue, Jun 30, 2020 at 1:06 PM Ralph Goers <[hidden email]>
wrote:

> If you can live with the GelfLayout you could also try that. It will let
> you format the message attribute using a Pattern String equivalent to the
> PatternLayout, so you use its expression to limit the length.
>
> Ralph
>
> > On Jun 30, 2020, at 12:28 PM, Volkan Yazıcı <[hidden email]>
> wrote:
> >
> > Hello Ayush,
> >
> > As of this moment, there doesn't exist such a configuration knob in
> > JsonLayout, AFAIK. Though the future is bright! There is an external
> > plugin, log4j2-logstash-layout[1], which delivers what you're looking
> > for (see its maxStringLength parameter) and more! The good news is, we
> > are in the process of merging this external plugin into the Log4j core
> > with a major overhaul and rebranding, JsonTemplateLayout.
> > Implementation is completed, but it is waiting for a final review on a
> > particular aspect by one of the core members, Ralph Goers.
> >
> > My personal advice would be to go with LogstashLayout and switch to
> > JsonTemplateLayout once it is there.
> >
> > Kind regards.
> >
> > [1] https://github.com/vy/log4j2-logstash-layout
> >
> > On Tue, Jun 30, 2020 at 8:47 PM Ayush Kumar <[hidden email]>
> wrote:
> >>
> >> Not sure where else to ask this question (stackoverflow maybe?), so
> >> apologies if this isn't the right forum.
> >>
> >> We've recently migrated to log4j2 from logback and are using the
> jsonlayout
> >> for a file appender. These logs are then parsed by scalyr (awesome tool
> >> btw). We've noticed that extended stacktraces in jsonlayout are
> >> ridiculously long. Reaching >20k characters, even with string mode
> enabled.
> >> This is a problem because there's a max log line size that we can parse
> >> (something around 8k I think).
> >>
> >> I was wondering if there was any way to get the default jsonlayout to
> cut
> >> down the size of stacktraces. PatternLayout has %ex{n}, and I was hoping
> >> for something that wasn't creating my own layout, rewriting every log
> >> message, or manually making pattern layout be a json format with
> %enc{json}
> >> and a pattern like {"message": "%enc(%m){json}", ...}.
> >>
> >> I've created a ticket for it as well
> >> https://issues.apache.org/jira/browse/LOG4J2-2879.
> >>
> >> Thanks.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>