Hi Sebastian,
Find output below ;
sha256sum /usr/lib/python3.5/csv.py
6d96e56ec22d1603ffed0f8d20f5b5090fb6ae4c7417126d09afc45447b34a64  /usr/lib/python3.5/csv.py


Below is error I get after I parse email;

2020-03-19 14:02:16,409 - ShadowServer-Parser - ERROR - Failed to parse line.
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 951, in process
    events = list(filter(bool, value))
  File "/usr/local/lib/python3.5/dist-packages/intelmq/bots/parsers/shadowserver/parser.py", line 111, in parse_line
    ' format or misconfiguration.'.format(shadowkey, self.feedname))
ValueError: Required column 'timestamp' not found in feed 'Blacklisted-IP'. Possible change in data format or misconfiguration.
2020-03-19 14:02:16,410 - ShadowServer-Parser - INFO - Dumping message to dump file.
2020-03-19 14:02:16,514 - ShadowServer-Parser - INFO - Dumping message to dump file.
2020-03-19 14:02:16,612 - ShadowServer-Parser - INFO - Sent 0 events and found 2 problem(s).


From Dump;
       "source_queue": "ShadowServer-Parser-queue",
        "traceback": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py\", line 951, in process\n    events = list(filter(bool, value))\n  File \"/usr/local/lib/python3.5/dist-packages/intelmq/bots/parsers/shadowserver/parser.py\", line 111, in parse_line\n    ' format or misconfiguration.'.format(shadowkey, self.feedname))\nValueError: Required column 'timestamp' not found in feed 'Blacklisted-IP'. Possible change in data format or misconfiguration.\n"
    }

Thank you,
Patrick

On Thursday, March 19, 2020, 01:48:02 PM GMT+3, Sebastian Wagner <wagner@cert.at> wrote:


Hi,

On a fresh, virgin and up-to-date xenial installation I get the same results as Bernhard, which differ from yours. I suggest to re-install the affected python3 package on your system (or, maybe just the whole python3-stack):

sudo apt-get install --reinstall libpython3.5-stdlib

And then show the output of the commands again.

Sebsatian
On 3/19/20 10:48 AM, patric sungura wrote:
Dear team,
Find below inputs as requested;

##ls -l /usr/lib/python3.5/csv.py
-rw-r--r-- 1 root root 16138 Mar 19 11:28 /usr/lib/python3.5/csv.py

##sha256sum /usr/lib/python3.5/csv.py
fbed6ea919f70b4fd29608ab9620785304fc50a4b90cd2dd5d2363a0b81d9e81  /usr/lib/python3.5/csv.py

Find line aroud line#96

 79 class DictReader:
 80     def __init__(self, f, fieldnames=None, restkey=None, restval=None,
 81                  dialect="excel", *args, **kwds):
 82         self._fieldnames = fieldnames   # list of keys for the dict
 83         self.restkey = restkey          # key to catch long rows
 84         self.restval = restval          # default value for short rows
 85         self.reader = reader(f, dialect, *args, **kwds)
 86         self.dialect = dialect
 87         self.line_num = 0
 88 
 89     def __iter__(self):
 90         return self
 91 
 92     @property
 93     def fieldnames(self):
 94         if self._fieldnames is None:
 95             try:
 96                 self._fieldnames = next(self.reader)
 97             except StopIteration:
 98                 pass
 99         self.line_num = self.reader.line_num
100         return self._fieldnames
101 
102     @fieldnames.setter
103     def fieldnames(self, value):
104         self._fieldnames = value 

I have two shadow server Parser and now have recorded different errors;

ShadowServer-Parser-2 
2020-03-19 12:31:07,318 - ShadowServer-Parser-2 - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start
    self.process()
  File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process
    for line in self.parse(report):
  File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict
    self.csv_fieldnames = csv_reader.fieldnames
  File "/usr/lib/python3.5/csv.py", line 96, in fieldnames
    self._fieldnames = next(self.reader)
NameError: name 'csv' is not defined
2020-03-19 12:31:07,319 - ShadowServer-Parser-2 - INFO - Dumping message to dump file.

ShadowServer-Parser
2020-03-19 12:37:58,573 - ShadowServer-Parser - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 267, in start
    self.process()
  File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 941, in process
    for line in self.parse(report):
  File "/usr/local/lib/python3.5/dist-packages/intelmq/lib/bot.py", line 884, in parse_csv_dict
    self.csv_fieldnames = csv_reader.fieldnames
  File "/usr/lib/python3.5/csv.py", line 96, in fieldnames
    self._fieldnames = next(self.reader)
_csv.Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?
2020-03-19 12:37:58,574 - ShadowServer-Parser - INFO - Dumping message to dump file



Thank you very much,
Patrick
On Thursday, March 19, 2020, 11:56:49 AM GMT+3, Bernhard Reiter <bernhard@intevation.de> wrote:


Am Mittwoch, 18. März 2020, 17:02:13 CET schrieben Sie:
>  libpython3.5-stdlib:amd64 3.5.2-2ubuntu0~16.04.9 amd64

This is the standard version, but still the backtrace looks strange,
especially the last bit:
  File "/usr/lib/python3.5/csv.py", line 96, in fieldnames
  self._fieldnames = next(self.reader, dialect=csv.excel_tab)
  NameError: name 'csv' is not defined

The second part is strange as the original sources [1] don't have the second
parameter. So we should double check.
Can you open the file `/usr/lib/python3.5/csv.py` and check line 96?

Please send the whole function around the lines here.
(Can you also post your mail in text-only format? This is something I can and
many others can read much faster, and thus can answer you faster. :) Thanks! )

Can you also post the length, permissions and sha256 sum of the file?
Here for comparison the data for my file:
ls -l /usr/lib/python3.5/csv.py
-rw-r--r-- 1 root root 16128 Oct  8 16:38 /usr/lib/python3.5/csv.py
sha256sum /usr/lib/python3.5/csv.py
6d96e56ec22d1603ffed0f8d20f5b5090fb6ae4c7417126d09afc45447b34a64  /usr/lib/
python3.5/csv.py

Best Regards,
Bernhard
[1] https://github.com/python/cpython/blob/master/Lib/csv.py


--
www.intevation.de/~bernhard  +49 541 33 508 3-3
Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner

-- 
// Sebastian Wagner <wagner@cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg