Hi Patrick,
Am Montag 01 Juni 2020 16:27:24 schrieb Patrick Forsberg:
> Anyone have a quick comparison between AbuseIO and IntelMQ?
>
> I'm currently in the process of getting IntelMQ to work with
> IntelMQ-Mailgen to be able to send out abuse-emails to our constituency
> based on feeds like Shadowserver and since it seems like AbuseIO can do
> just about the same I would like to know the pros and cons of the systems.
for comparing AbuseIO and IntelMQ, I don't know AbuseIO enough.
However if you are looking into IntelMQ Mailgen from
the system we call intelmq-cb-mailgen,
https://github.com/Intevation/intelmq-mailgen-release
that is the one we've been developing for the CERT-Bund,
so we can tell you more about it, maybe this helps with the comparison.
The design idea is to be automated, flexible and high through-put.
Thus there is a separation of concerns and several configuration places.
You may have seen the overview diagram:
https://raw.githubusercontent.com/Intevation/intelmq-mailgen/master/docs/no…
There are rules within the IntelMQ export and additional notification formats
scripts, those are quite flexible, so there is some learning curve.
Once set up, there can be millions of events handled per day automatically
with several people being able to add manual data to the contacts database.
Feel free to ask us here or per direct email, if you have any questions or
need help with setting it up.
Best Regards,
Bernhard
--
www.intevation.de/~bernhard +49 541 33 508 3-3
Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
Dear developers and pro-users,
I have pushed a 2.2.0 pre-release to both PyPI as well as the unstable
repository. If you have time & resources, or you are using the develop
branch anyway, please test this version so we can ship the next stable
release ready next week or the week afterwards, depending on the feedback.
Please note that the current codebase is no longer tested with Python
3.4 and it may not work anymore with that Python version.
For installation/upgrade with pip use the --pre parameter: pip install
--pre intelmq
Instructions for the deb & rpm unstable repository:
https://software.opensuse.org/download/package?package=intelmq&project=home…
best regards
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
We already collected a very long list of bug fixes since the last
release, so it was time to mark the next milestone! As usual, you can
find the list of changes below. The pre-built deb/rpm packages will hit
the repositories very soon.
Installation documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.1.3/docs/UPGRADING.md
Full changelog:
### Requirements
- The python library `requests` is (again) listed as dependency of the
core (#1519).
### Core
- `intelmq.lib.upgrades`:
- Harmonization upgrade: Also check and update regular expressions.
- Add function to migrate the deprecated parameter `attach_unzip` to
`extract_files` for the mail attachment collector.
- Add function to migrate changed Taichung URL feed.
- Check for discontinued Abuse.CH Zeus Tracker feed.
- `intelmq.lib.bot`:
- `ParserBot.recover_line`: Parameter `line` needs to be optional, fix
usage of fallback value `self.current_line`.
- `start`: Handle decoding errors in the pipeline different so that
the bot is not stuck in an endless loop (#1494).
- `start`: Only acknowledge a message in case of errors, if we
actually had a message to dump, which is not the case for collectors.
- `_dump_message`: Dump messages with encoding errors base64 encoded,
not in JSON format as it's not possible to decode them (#1494).
- `intelmq.lib.test`:
- `BotTestCase.run_bot`: Add parameters `allowed_error_count` and
`allowed_warning_count` to allow set the number per run, not per test class.
- Set `source_pipeline_broker` and `destination_pipeline_broker` to
`pythonlist` instead of the old `broker`, fixes
`intelmq.tests.lib.test_bot.TestBot.test_pipeline_raising`.
- Fix test for (allowed) errors and warnings.
- `intelmq.lib.exceptions`:
- `InvalidKey`: Add `KeyError` as parent class.
- `DecodingError`: Added, string representation has all relevant
information on the decoding error, including encoding, reason and the
affected string (#1494).
- `intelmq.lib.pipeline`:
- Decode messages in `Pipeline.receive` not in the implementation's
`_receive` so that the internal counter is correct in case of decoding
errors (#1494).
- `intelmq.lib.utils`:
- `decode`: Raise new `DecodingError` if decoding fails.
### Harmonization
- `protocol.transport`: Adapt regular expression to allow the value
`nvp-ii` (protocol 11).
### Bots
#### Collectors
- `intelmq.bots.collectors.mail.collector_mail_attach`:
- Fix handling of deprecated parameter name `attach_unzip`.
- Fix handling of attachments without filenames (#1538).
- `intelmq.bots.collectors.stomp.collector`: Fix compatibility with
stomp.py versions `> 4.1.20` and catch errors on shutdown.
- `intelmq.bots.collectors.microsoft`:
- Update `REQUIREMENTS.txt` temporarily fixing deprecated Azure
library (#1530, PR#1532).
- `intelmq.bots.collectors.microsoft.collector_interflow`: Add method
for printing the file list.
#### Parsers
- `intelmq.bots.parsers.cymru.parser_cap_program`: Support for protocol
11 (`nvp-ii`) and `conficker` type.
- `intelmq.bots.parsers.taichung.parser`: Support more
types/classifications:
- Application Compromise: Apache vulnerability & SQL injections
- Brute-force: MSSQL & SSH password guess attacks; Office 365, SSH &
SIP attacks
- C2 Sever: Attack controller
- DDoS
- DoS: DNS, DoS, Excess connection
- IDS Alert / known vulnerability exploitation: backdoor
- Malware: Malware Proxy
- Warn on new unknown types.
- `intelmq.bots.parsers.bitcash.parser`: Removed as feed is discontinued.
- `intelmq.bots.parsers.fraunhofer.parser_ddosattack_cnc` and
`intelmq.bots.parsers.fraunhofer.parser_ddosattack_target`: Removed as
feed is discontinued.
- `intelmq.bots.parsers.malwaredomains.parser`: Correctly classify `C&C`
and `phishing` events.
- `intelmq.bots.parsers.shadowserver.parser`: More verbose error message
for missing report specification (#1507).
- `intelmq.bots.parsers.n6.parser_n6stomp`: Always add n6 field `name`
as `malware.name` independent of `category`.
- `intelmq.bots.parsers.anubisnetworks`: Update parser with new data format.
- `intelmq.bots.parsers.bambenek`: Add new feed URLs with Host
`faf.bambenekconsulting.com` (#1525, PR#1526).
- `intelmq.bots.parsers.abusech.parser_ransomware`: Removed, as the feed
is discontinued (#1537).
- `intelmq.bots.parsers.nothink.parser`: Removed, as the feed is
discontinued (#1537).
- `intelmq.bots.parsers.n6.parser`: Remove not allowed characters in the
name field for `malware.name` and write original value to
`event_description.text` instead.
#### Experts
- `intelmq.bots.experts.cymru_whois.lib`: Fix parsing of AS names with
Unicode characters.
#### Outputs
- `intelmq.bots.outputs.mongodb`:
- Set default port 27017.
- Use different authentication mechanisms per MongoDB server version
to fix compatibility with server version >= 3.4 (#1439).
### Documentation
- Feeds:
- Remove unavailable feed Abuse.CH Zeus Tracker.
- Remove the field `status`, offline feeds should be removed.
- Add a new field `public` to differentiate between private and public
feeds.
- Adding documentation URLs to nearly all feeds.
- Remove unavailable Bitcash.cz feed.
- Remove unavailable Fraunhofer DDos Attack feeds.
- Remove unavailable feed Abuse.CH Ransomware Tracker (#1537).
- Update information on Bambenek Feeds, many require a license now
(#1525).
- Remove discontinued Nothink Honeypot Feeds (#1537).
- Developers Guide: Fix the instructions for `/opt/intelmq` file
permissions.
### Packaging
- Patches: `fix-logrotate-path.patch`: also include path to rotated file
in patch.
- Fix paths from `/opt` to LSB for `setup.py` and
`contrib/logrotate/intelmq` in build process (#1500).
- Add runtime dependency `debianutils` for the program `which`, which is
required for `intelmqctl`.
### Tests
- Dropping Travis tests for 3.4 as required libraries dropped 3.4 support.
- `intelmq.tests.bots.experts.cymru_whois`:
- Drop missing ASN test, does not work anymore.
- IPv6 to IPv4 test: Test for two possible results.
- `intelmq.lib.test`: Fix compatibility of logging capture with Python
>= 3.7 by reworking the whole process (#1342).
- `intelmq.bots.collectors.tcp.test_collector`: Removing custom mocking
and bot starting, not necessary anymore.
- Added tests for
`intelmq.bin.intelmqctl.IntelMQProcessManager._interpret_commandline`.
- Fix and split
`tests.bots.experts.ripe.test_expert.test_ripe_stat_error_json`.
- Added tests for invalid encodings in input messages in
`intelmq.tests.lib.test_bot` and `intelmq.tests.lib.test_pipeline` (#1494).
- Travis: Explicitly enable RabbitMQ management plugin.
- `intelmq.tests.lib.test_message`: Fix usage of the parameter
`blacklist` for Message hash tests (#1539).
### Tools
- `intelmqsetup`: Copy missing BOTS file to IntelMQ's root directory
(#1498).
- `intelmq_gen_docs`: Feed documentation generation: Handle
missing/empty parameters.
- `intelmqctl`:
- `IntelMQProcessManager`: For the status of running bots also check
the bot ID of the commandline and ignore the path of the executable (#1492).
- `IntelMQController`: Fix exit codes of `check` command for JSON
output (now 0 on success and 1 on error, was swapped, #1520).
- `intelmqdump`:
- Handle base64-type messages for show, editor and recovery actions.
### Contrib
- `intelmq/bots/experts/asn_lookup/update-asn-data`: Use
`pyasn_util_download.py` to download the data instead from RIPE, which
cannot be parsed currently (#1517, PR#1518,
https://github.com/hadiasghari/pyasn/issues/62).
### Known issues
- HTTP stream collector: retry on regular connection problems? (#1435).
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952).
- Reverse DNS: Only first record is used (#877).
- Corrupt dump files when interrupted during writing (#870).
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi Sebastian,
do you have a rough timeframe for the 2.2.0 release?
Best,
Bernhard
--
www.intevation.de/~bernhard +49 541 33 508 3-3
Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
Hi IntelMQ-Devs,
wondering what an OutputBot should do, if it cannot
put an event to the output, because of the event itself.
Should it do something like
self.logger.warning("event does not meet criteria for output")
self._dump_message()
# to place the event in a dump file for later inspection
self.acknowledge_message()
?
Background: the
https://github.com/certtools/intelmq/blob/develop/intelmq/bots/outputs/misp…
seems to get events that it cannot insert into MISP, because
some fields necessary in the intelmq event are not filled with values.
If the bot detects this, it can skip the event,
but it seems a good idea to preserve enough info how the empty values came to
be.
The alternatives to dumping would be
a) write out the event in the log using self.logger
b) just ignore the event
Thanks,
Bernhard
--
www.intevation.de/~bernhard +49 541 33 508 3-3
Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
Dear IntelMQ community,
I hope everyone is fine and most of us are slowly waking up after the Corona lock-down phase ( at least I am, and it's a really good feeling to be able to work concentrated on things again. Let me tell you, home-office with small children is not fun).
In any case, I already had some telcos with some of you regarding our wish list for IntelMQ-3.0. There are some fantastic ideas floating around and since IntelMQ is such a versatile and adaptable tool, it is quite a challenge to capture all the great changes and adaptations people made to it.
The next version of IntelMQ (3.0) will try to capture some of the most needed changes (and some wishes) for you.
I tried to summarise some of these changes in the following document:
https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architectu…
May I ask you to review it and send me feedback (kaplan(a)cert.at)? Also, I'd be very happy to have a telco with you in case that's easier or more natural for you for giving feedback.
Since we would like to start coding soon, it is important to receive *your* input on the next version of IntelMQ.
Thanks for your time!
All the best,
Aaron.
--
// L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - http://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
The newest IntelMQ Manager release 2.1.1 fixes a critical security bug.
Please never run the IntelMQ Manager without proper authentication in place!
Installation instructions:
https://github.com/certtools/intelmq-manager/blob/2.1.1/docs/INSTALL.md
Bernhard Herzog (Intevation) discovered that the backend incorrectly
handled messages given by user-input in the "send" functionality of the
Inspect-tool of the Monitor component. An attacker with access to the
IntelMQ Manager could possibly use this issue to execute arbitrary code
with the privileges of the webserver.
Updated deb/rpm-packages are already available in the repositories.
Other changes:
### Backend
- Fix misspelling of the environmental variable
`INTELMQ_MANGER_CONTROLLER_CMD` to `INTELMQ_MANAGER_CONTROLLER_CMD` (an
'a' was missing).
- Fix handling of POST variable `msg` of the message-sending
functionality available in the Inspect-tool.
### Pages
#### Monitor
- Fix running commands with the "inspect" widget by fixing the
definition of the `CONTROLLER_CMD` in the template (PR #194).
### Documentation
- Update supported operating systems in Installation documentation (i.a.
PR #191).
### Known issues
* Missing CSRF protection (#111).
* Graph jumps around on "Add edge" (#148).
* wrong error message for new bots with existing ID (#152).
* `ALLOWED_PATH=` violates CSP (#183).
* Monitor page: Automatic log refresh reset log page to first one (#190).
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
The ElasticSearch bots, tests and tools in IntelMQ need some maintenance
which I am unable to provide. As ES is a very common tool I am sure that
there is know-how available in the community and we are able to continue
the support for it.
The oldest know issue is a broken unittest:
https://github.com/certtools/intelmq/issues/1480
But there are also incompatibilties with current ElasticSearch version,
e.g. I had problems with the elasticmapper tool using ES 7.6.1 (maybe
easy to fix).
Using 7.5.0 failed on the indices tests
https://github.com/certtools/intelmq/issues/1479
Further, the only supported elasticsearch python library version is
currently 'elasticsearch>=5.0.0,<6.0.0' while the latest release is 7.6.0.
Please consider contributing
best regards
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Hi,
(involving intelmq-dev now so that we can move that discussion to the
developers' list)
Thanks for the hint. That could be a possible replacement. When
analyzing the stat.ripe.net webinterface I also found this endpoint
giving the same result:
https://stat.ripe.net/data/rir-geo/data.json?resource=131.130.254.77
Has anyone a clue why RIPE provides so many different endpoints for the
same data? (With different status which is not properly propagated to
the status code...)
best wishes,
Sebastian
On 3/17/20 10:55 PM, Chris Horsley wrote:
>
> Is this alternative RIPE API endpoint a feasible alternative?
>
> https://stat.ripe.net/data/geoloc/data.json?resource=131.130.254.77/24
>
> Cheers,
>
> Chris
>
> On 18/03/2020 3:14 am, Sebastian Wagner wrote:
>>
>> Hi,
>>
>> I just noticed, that RIPE currently does not provide geolocation
>> information anymore as a result of the MaxMind data license change.
>> That data can/could be queried with the IntelMQ RIPE expert. In case
>> you are still relying on this information, please use another source
>> for geolocation data, like the maxmind geolocation expert and local
>> data. Unfortunately, the returned status code of the API call is 200
>> and the error is only detectable by another field. I am working on
>> changes in the RPIE expert to detect this and raise a warning for it.
>>
>> best regards,
>> Sebastian
>>
>> For example
>> https://stat.ripe.net/data/maxmind-geo-lite/data.json?resource=131.130.254.…
>> says:
>>
>> messages
>> 0
>> 0 "info"
>> 1 "This data is currently unavailable due to maintenance. Please
>> check official announcements for when it will be available again!
>> https://stat.ripe.net/feedback"
>> data_call_status "maintenance - this data call is in maintenance mode"
>>
>> --
>> // Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
>> // CERT Austria - https://www.cert.at/
>> // Eine Initiative der nic.at GmbH - https://www.nic.at/
>> // Firmenbuchnummer 172568b, LG Salzburg
>>
>
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
