使用processor_split_log_regex偶尔会不正确切分 #828

q121102302 · 2023-05-04T12:50:43Z

q121102302
May 4, 2023

使用版本：1.3.1
配置：
ilogtail_config.json

user_yaml_config.d下的配置文件

需要获取的日志：

日志流程是 ilogtail->kafka->java->es
在es发现会有数据偶尔不是正常切分的
比如上面这条在es中是两条数据因为我们需要做一些正则额外获取数据就导致下面这条没有大部分信息从而很难搜索到

Answered by Takuka0311

Aug 13, 2024

新版本是指2.X的版本吗，如果是1.X的版本这个问题是都存在的吗，不太方便直接升级的情况下有没有什么替代方案之类的

1.8.x版本已经是C++部分切分。1.8.x以上版本也可以使用C++实现的原生处理插件（比如processor_regex_native等），性能更好

View full answer

messixukejia · 2023-05-04T13:09:35Z

messixukejia
May 4, 2023
Maintainer

从时间看并不是一条日志

0 replies

q121102302 · 2023-05-05T01:51:10Z

q121102302
May 5, 2023
Author

不好意思当时这个一直在报这个相同的错误就直接截取了一个附近的
这是257那一毫秒的文件日志共出现三次warn

这是es写入的数据查询出来也有三次warn
`{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 3,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "ilogtail-inclution-log-2023-05-04",
"_type" : "_doc",
"_id" : "l6K65ocBtTQeWfp_An9M",
"_score" : null,
"_source" : {
"app" : "micro-scm-mes",
"channel" : "default",
"content" : """
2023-05-04 20:26:01.257 [TID:N/A] WARN [Thread-21] o.s.c.a.AnnotationConfigApplicationContext Line:994 - Exception thrown from ApplicationListener handling ContextClosedEvent
org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'eurekaAutoServiceRegistration': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:216)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1080)
at org.springframework.context.event.ApplicationListenerMethodAdapter.getTargetBean(ApplicationListenerMethodAdapter.java:283)
at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:253)
at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:177)
at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:140)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:399)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:347)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:991)
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:958)
at org.springframework.cloud.context.named.NamedContextFactory.destroy(NamedContextFactory.java:76)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:272)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:961)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.destroySingletons(FactoryBeanRegistrySupport.java:230)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:968)
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1032)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1008)
at org.springframework.context.support.AbstractApplicationContext$2.run(AbstractApplicationContext.java:929)

""",
"env" : "prod-pt",
"host" : "hzk-prod-k8s2-n3",
"ip" : "172.16.89.0",
"level" : "warn",
"loggerName" : "o.s.c.a.AnnotationConfigApplicationContext",
"threadName" : "Thread-21",
"tid" : "N/A",
"time" : 1683203161257
},
"sort" : [
1683203161257
]
},
{
"_index" : "ilogtail-inclution-log-2023-05-04",
"_type" : "_doc",
"_id" : "kqK65ocBtTQeWfp_An9M",
"_score" : null,
"_source" : {
"app" : "micro-scm-mes",
"channel" : "default",
"content" : """
2023-05-04 20:26:01.257 [TID:N/A] WARN [Thread-21] o.s.c.a.AnnotationConfigApplicationContext Line:994 - Exception thrown from ApplicationListener handling ContextClosedEvent
org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'eurekaAutoServiceRegistration': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:216)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1080)
at org.springframework.context.event.ApplicationListenerMethodAdapter.getTargetBean(ApplicationListenerMethodAdapter.java:283)
at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:253)
at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:177)
at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:140)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
""",
"env" : "prod-pt",
"host" : "hzk-prod-k8s2-n3",
"ip" : "172.16.89.0",
"level" : "warn",
"loggerName" : "o.s.c.a.AnnotationConfigApplicationContext",
"threadName" : "Thread-21",
"tid" : "N/A",
"time" : 1683203161257
},
"sort" : [
1683203161257
]
},
{
"_index" : "ilogtail-inclution-log-2023-05-04",
"_type" : "_doc",
"_id" : "laK65ocBtTQeWfp_An9M",
"_score" : null,
"_source" : {
"app" : "micro-scm-mes",
"channel" : "default",
"content" : """
2023-05-04 20:26:01.257 [TID:N/A] WARN [Thread-21] o.s.c.a.AnnotationConfigApplicationContext Line:994 - Exception thrown from ApplicationListener handling ContextClosedEvent
org.springframework.beans.factory.BeanCreationNotAllowedException: Error creating bean with name 'eurekaAutoServiceRegistration': Singleton bean creation not allowed while singletons of this factory are in destruction (Do not request a bean from a BeanFactory in a destroy method implementation!)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:216)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1080)
at org.springframework.context.event.ApplicationListenerMethodAdapter.getTargetBean(ApplicationListenerMethodAdapter.java:283)
at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:253)
at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:177)
at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:140)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:399)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:347)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:991)
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:958)
at org.springframework.cloud.context.named.NamedContextFactory.destroy(NamedContextFactory.java:76)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:272)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:961)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.destroySingletons(FactoryBeanRegistrySupport.java:230)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:968)
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1032)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1008)
at org.springframework.context.support.AbstractApplicationContext$2.run(AbstractApplicationContext.java:929)

""",
"env" : "prod-pt",
"host" : "hzk-prod-k8s2-n3",
"ip" : "172.16.89.0",
"level" : "warn",
"loggerName" : "o.s.c.a.AnnotationConfigApplicationContext",
"threadName" : "Thread-21",
"tid" : "N/A",
"time" : 1683203161257
},
"sort" : [
1683203161257
]
}
]
}
}
`

这是额外出来的一条数据
`{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "ilogtail-inclution-log-1970-01-20",
"_type" : "_doc",
"_id" : "k6K65ocBtTQeWfp_An9M",
"_score" : null,
"_source" : {
"app" : "micro-scm-mes",
"channel" : "default",
"content" : """
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:393)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:399)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:347)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:991)
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:958)
at org.springframework.cloud.context.named.NamedContextFactory.destroy(NamedContextFactory.java:76)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:272)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:961)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523)
at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.destroySingletons(FactoryBeanRegistrySupport.java:230)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:968)
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1032)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1008)
at org.springframework.context.support.AbstractApplicationContext$2.run(AbstractApplicationContext.java:929)

""",
"env" : "prod-pt",
"host" : "hzk-prod-k8s2-n3",
"ip" : "172.16.89.0",
"time" : 1683203161
},
"sort" : [
1683203161
]
}
]
}
}
`

可以看出应该是第二条warn日志没有正确的进行多行切分

0 replies

messixukejia · 2023-05-05T02:07:20Z

messixukejia
May 5, 2023
Maintainer

把原始日志上下的文本及采集配置的文本发下吧。

0 replies

q121102302 · 2023-05-05T09:45:18Z

q121102302
May 5, 2023
Author

辫辫的yaml和ilogtail_config文件：
file_kafka.txt
ilogtail_config.txt
这是那部分日志文件：
micro-scm-mes-info-2023-05-04.0-20230505112056364414.log

3 replies

yyuuttaaoo May 5, 2023
Maintainer

这个看上去有点像一个已知问题：使用插件时，多行日志切分有时会丢失后半段内容。日志量比较大，读取的时候有多行日志跨buffer的情况就会触发bug

q121102302 May 6, 2023
Author

那这个会有解决方案吗。。。因为场景设计日志要做一些事情异常截断的话信息会不全

yyuuttaaoo Jun 21, 2023
Maintainer

这个问题我们会将多行切分全部放到C++处理，同统一解决多行切分的几个问题。

yespp · 2024-08-12T03:30:08Z

yespp
Aug 12, 2024

这个问题解决了吗，我也遇到一样的情况

3 replies

Takuka0311 Aug 13, 2024
Collaborator

现在最新版本的ilogtail，多行切分已经放在了C++部分进行，不需要额外配置即可生效

yespp Aug 13, 2024

新版本是指2.X的版本吗，如果是1.X的版本这个问题是都存在的吗，不太方便直接升级的情况下有没有什么替代方案之类的

Takuka0311 Aug 13, 2024
Collaborator

新版本是指2.X的版本吗，如果是1.X的版本这个问题是都存在的吗，不太方便直接升级的情况下有没有什么替代方案之类的

1.8.x版本已经是C++部分切分。1.8.x以上版本也可以使用C++实现的原生处理插件（比如processor_regex_native等），性能更好

Answer selected by Takuka0311

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

使用processor_split_log_regex偶尔会不正确切分 #828

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

使用processor_split_log_regex偶尔会不正确切分 #828

q121102302 May 4, 2023

Replies: 5 comments · 6 replies

messixukejia May 4, 2023 Maintainer

q121102302 May 5, 2023 Author

messixukejia May 5, 2023 Maintainer

q121102302 May 5, 2023 Author

yyuuttaaoo May 5, 2023 Maintainer

q121102302 May 6, 2023 Author

yyuuttaaoo Jun 21, 2023 Maintainer

yespp Aug 12, 2024

Takuka0311 Aug 13, 2024 Collaborator

yespp Aug 13, 2024

Takuka0311 Aug 13, 2024 Collaborator

q121102302
May 4, 2023

Replies: 5 comments 6 replies

messixukejia
May 4, 2023
Maintainer

q121102302
May 5, 2023
Author

messixukejia
May 5, 2023
Maintainer

q121102302
May 5, 2023
Author

yyuuttaaoo May 5, 2023
Maintainer

q121102302 May 6, 2023
Author

yyuuttaaoo Jun 21, 2023
Maintainer

yespp
Aug 12, 2024

Takuka0311 Aug 13, 2024
Collaborator

Takuka0311 Aug 13, 2024
Collaborator