Logstash收集CSV日志解析CSV文件示例：

阿里的示例:

https://help.aliyun.com/document_detail/28994.html

使用系统时间作为日志时间上传日志样例10.116.14.201,-,2/25/2016,11:53:17,W3SVC7,2132,200,0,GET,project/shenzhen-test/logstore/logstash/detail,C:\test\csv\test_csv.log
采集配置input {file {type &＃61;> "csv_log_1"path &＃61;> ["C:/test/csv/*.log"]start_position &＃61;> "beginning"}
}
filter {if [type] &＃61;&＃61; "csv_log_1" {csv {separator &＃61;> ","columns &＃61;> ["ip", "a", "date", "time", "b", "latency", "status", "size", "method", "url", "file"]} }
}
output {if [type] &＃61;&＃61; "csv_log_1" {logservice {codec &＃61;> "json"endpoint &＃61;> "***"project &＃61;> "***"logstore &＃61;> "***"topic &＃61;> ""source &＃61;> ""access_key_id &＃61;> "***"access_key_secret &＃61;> "***"max_send_retry &＃61;> 10}}
}
注意&＃xff1a;配置文件格式必须以 UTF-8 无 BOM 格式编码&＃xff0c;可以下载 notepad&＃43;&＃43; 修改文件编码格式。
path 填写文件路径时请使用 UNIX 模式的分隔符&＃xff0c;如&＃xff1a;C:/test/multiline/*.log&＃xff0c;否则无法支持模糊匹配。
type 字段需要统一修改并在该文件内保持一致&＃xff0c;如果单台机器存在多个Logstash配置文件&＃xff0c;需要保证各配置 type 字段唯一&＃xff0c;否则会导致数据处理的错乱。
相关插件&＃xff1a;file、csv。重启Logstash生效创建配置文件到 conf 目录&＃xff0c;参考配置Logstash重启Logstash生效。使用日志字段内容作为日志时间上传日志样例10.116.14.201,-,Feb 25 2016 14:03:44,W3SVC7,1332,200,0,GET,project/shenzhen-test/logstore/logstash/detail,C:\test\csv\test_csv_withtime.log
采集配置input {file {type &＃61;> "csv_log_2"path &＃61;> ["C:/test/csv_withtime/*.log"]start_position &＃61;> "beginning"}
}
filter {if [type] &＃61;&＃61; "csv_log_2" {csv {separator &＃61;> ","columns &＃61;> ["ip", "a", "datetime", "b", "latency", "status", "size", "method", "url", "file"]} date {match &＃61;> [ "datetime" , "MMM dd YYYY HH:mm:ss" ]}}
}
output {if [type] &＃61;&＃61; "csv_log_2" {logservice {codec &＃61;> "json"endpoint &＃61;> "***"project &＃61;> "***"logstore &＃61;> "***"topic &＃61;> ""source &＃61;> ""access_key_id &＃61;> "***"access_key_secret &＃61;> "***"max_send_retry &＃61;> 10}}
}
注意&＃xff1a;配置文件格式必须以 UTF-8 无 BOM 格式编码&＃xff0c;可以下载 notepad&＃43;&＃43; 修改文件编码格式。
path 填写文件路径时请使用 UNIX 模式的分隔符&＃xff0c;如&＃xff1a;C:/test/multiline/*.log&＃xff0c;否则无法支持模糊匹配。
type 字段需要统一修改并在该文件内保持一致&＃xff0c;如果单台机器存在多个Logstash配置文件&＃xff0c;需要保证各配置 type 字段唯一&＃xff0c;否则会导致数据处理的错乱。

字段&＃xff1a;

"Version","CsvFileFormatVersion","OutputInfo","OutputInfoFaultComponentInfo","OutputInfoPinInfo","InspectionMachine","InspectionProcess","ProgramName ","BothSideCode","CreateMachine","LibraryName","ReferencePosition","PcbSizeHeight","PcbSizeWidth","RailWidth","SavedDate","PcbNo","TestTime","CreateDate","PersonRevisor","PersonRevisorCode","ReviseTime","ReviseEndDate","TestResult","ReviseResult","OverlookFault","LotCount","Barcode","FaultRate","ComponentTotal","PinTotal","LandTotal","OutComponentTotal","VisualFaultFlag","RevisorMachineId","RevisorComputerName","Barcode","BothSideCode","PcbNo","ComponentBlockNo","ComponentBlockName","ComponentNo","PartsName","PartsTypeNo","PartsVarNo","PartsVarName","PartsArticleNo","PinNo","PinSpaceNo","FaultCode","RevisedFaultId","XShift","YShift","AngleMount","ComponentReviseEndDate","OutComponentFlag","FaultLandCount","Grid","ComponentPersonRevisor","ComponentReviseTime","ComponentReviseMachineId","ComponentReviserComputerName"

input {file {path &＃61;> ["/opt/logstash-6.0.0/config/csv/*.csv"]start_position &＃61;> "beginning"}
}
filter {csv {separator &＃61;> ","columns &＃61;> ["Version","CsvFileFormatVersion","OutputInfo","OutputInfoFaultComponentInfo","OutputInfoPinInfo","InspectionMachine","InspectionProcess","ProgramName ","BothSideCode","CreateMachine","LibraryName","ReferencePosition","PcbSizeHeight","PcbSizeWidth","RailWidth","SavedDate","PcbNo","TestTime","CreateDate","PersonRevisor","PersonRevisorCode","ReviseTime","ReviseEndDate","TestResult","ReviseResult","OverlookFault","LotCount","Barcode","FaultRate","ComponentTotal","PinTotal","LandTotal","OutComponentTotal","VisualFaultFlag","RevisorMachineId","RevisorComputerName","Barcode","BothSideCode","PcbNo","ComponentBlockNo","ComponentBlockName","ComponentNo","PartsName","PartsTypeNo","PartsVarNo","PartsVarName","PartsArticleNo","PinNo","PinSpaceNo","FaultCode","RevisedFaultId","XShift","YShift","AngleMount","ComponentReviseEndDate","OutComponentFlag","FaultLandCount","Grid","ComponentPersonRevisor","ComponentReviseTime","ComponentReviseMachineId","ComponentReviserComputerName"]}
}
output {stdout{codec&＃61;>rubydebug}
}
~