Hello IntelMQ-Users,
if you are responsible to only deal with reports for a country
and base your decisions on the RIPE database,
how do you deal with more specific CIDRs that are from a different country,
but within a CIDR that belongs to yours?
See more details of the problem
as seen from the ripe importer the intelmq-cb-mailgen solution uses:
https://github.com/Intevation/intelmq-certbund-contact/issues/13
(Feel free to answer here or in the issue or personally.)
Thanks in advance,
Bernhard
--
www.intevation.de/~bernhard +49 541 33 508 3-3
Intevation GmbH, Osnabrück, DE; Amtsgericht Osnabrück, HRB 18998
Geschäftsführer Frank Koormann, Bernhard Reiter, Dr. Jan-Oliver Wagner
Hallo there,
I'm new to IntelMQ. But nevertheless I'm wondering how you are getting IntelMQ to use files uploaded in a directory. My only knowledge of IntelMQ Is using the GUI I'm not a scripting guy at all.
Could someone in this community please help with setting the right parameters for the "file-collector2 bot in the GUI of IntelMQ?
In advance many thanks.
Bedste hilsner/regards
Henrik Jensen
TeleDCIS
Tlf: +45 35 88 82 84
Mobil: +45 93 51 00 03
Mail: hj(a)teledcis.dk<mailto:hj@teledcis.dk>
www.teledcis.dk
Hi Everybody,
Could anyone of you lovely persons explain to me, what the following warning means and how to circumvent it?
Bedste hilsner/regards
Henrik Jensen
TeleDCIS
Tlf: +45 35 88 82 84
Mobil: +45 93 51 00 03
Mail: hj(a)teledcis.dk<mailto:hj@teledcis.dk>
www.teledcis.dk
Dear community,
TLDR: We are thinking about dropping support for Python 3.5 with the
release of IntelMQ 2.3.
At the moment IntelMQ supports Python back to and including Python 3.5,
which was released in 2015. The last version of Python 3.5 was 3.5.10
which was released this September and with that Python 3.5 has reached
end-of-life. The two Distributions that ship Python 3.5 only are Debian
9 and Ubuntu 16.04.
Debian 9 (Stretch) was superseded by Debian 10 (Buster) this summer, but
does still get LTS support until June 2022. Support for Ubuntu 16.04 LTS
(Xenial Xerus) will end in April 2021.
Therefore dropping support for Python 3.5 would mean that we would also
have to drop support for Debian 9 and Ubuntu 16.04.
The upside would not only be cleaner code through improvements in Python
3.6 but we could also update the packaging code. Furthermore, there
might be IntelMQ dependencies that drop support for Python 3.5 now that
it is EOL which would limit the functionality of IntelMQ.
So the basic question is, are there still (m)any setups out there that
require IntelMQ to run with Python 3.5?
cheers,
Birger
--
// Birger Schacht <schacht(a)cert.at>
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear IntelMQ developers and users,
below are a couple of ideas how to (hopefully) make configuration of
IntelMQ easier. Feel free to give feedback, voice concerns or simply ask
if there is something unclear. We plan to evaluate the feedback that
emerged in two weeks (after the christmas holidays).
# IntelMQ Configuration Handling (IntelMQ Enhancement Proposal 01)
## Format
### JSON
At the moment, the configuration format of IntelMQ is JSON[^1]. It
is parsed using the Python json library, which is part of the Python
Standard Library. The downside of JSON is, that is is hard to read
and and write for humans and it cannot contain comments.
[^1]: https://docs.python.org/3/library/index.html
### YAML
There is a proposal[^2] to use YAML as the default configuration
format. YAML provides way better readability for humans and YAML
supports single line comments. There are two Python YAML libraries
out there, the one being PyYAML[^3] and the other being
ruamel.yaml[^4]. The former is a project by the YAML project itself.
The latter is a fork of the former and had much more activity over
the years and better support of the standard. It seems that pyyaml
caught up in the last few years. We don't need any edge cases, so
both libraries would be good for configuration files. According to
this issue[^5] pyyaml does not support “editing YAML whilst
maintaining comments”, which might be a deal breaker, but this issue
is from 2016, this might have changed. On the other hand, IntelMQ
does not edit configuration at the moment. pyyaml and ruamel.yaml
are available as package in all relevant Linux distributions.
[^2]:
https://github.com/gethvi/intelmq/blob/ideas/docs/Ideas.md#changing-configu…
[^3]: https://pyyaml.org/
[^4]: https://yaml.readthedocs.io/en/latest/
[^5]: https://github.com/yaml/pyyaml/issues/46
### INI
The Python Standard Library also ships configparser[^6], which is a
“configuration language which provides a structure similar to what’s
found in Microsoft Windows INI files”. The files can contain
comments, it comes with a [DEFAULT] section, which can be used for
default values and the configuration files can contain variables.
One downside is that all the configurations are Strings, which means
we would have to do parsing ourself.
[^6]: https://docs.python.org/3/library/configparser.html
### toml
Tom's Obvious, Minimal Language is another contender for the role of
IntelMQs configuration file format. It looks similar to the INI file
format, but comes with various data types. It also allows comments.
There is a Python library[^7] that seems to be very active. toml is
also used as the format for the proposed pyproject.toml file and by
the rust community for their package configuration files. toml's
syntax for dictionaries is hard to read/write, harder than with
JSON.
[^7]: https://pypi.org/project/toml/
### Further information
* The summary on file formats on the PEP518 proposition:
https://www.python.org/dev/peps/pep-0518/#other-file-formats
* At the moment we are leaning towards YAML. Regarding the library,
we would choose ruamel.yaml, because it seems to have a more
active upstream and it can retain comments when it modifies a yaml
file.
## Storage
This part is about the question where do we store the
configuration?.
The ideas document[^8] on GitHub already proposes to remove the
pipeline.conf and specifying the destination pipelines in the
individual bot configuration part. The declaration of the source
queue can be dropped then as well, as it follows a rule anyway.
In addition to that, to make the setup of IntelMQ easier, the
defaults.conf should be dropped. Default values should be set in the
Bot classes respectively in the IntelMQ process managers, but there
is no need for a separate file.
Another question is, if every bot should have their own
configuration file. Some users wish to be able to start a bot
without having to rely on IntelMQ, but at the moment, the bot gets
the configuration from IntelMQ's runtime.conf. If we want to
support the request to be able to pass individual configurations to
bots, we could allow users to pass a separate configuration file to
the bot (i.e. using `-c /path/to/config.$ext`). If that file is not
set or does not contain the bots id, it is ignored and IntelMQ's
runtime.conf is used as usual. If it does exists, the global
runtime.conf is still parsed (if it exists - it should also be
possible to run a bot without a runtime.conf) but only the values
that are not set in the individual configuration file are
considered. This individual configuration file would also allow a
bot to be run in a docker environment without having to set any
environment variables. This would make configuration handling
probably easier, because then configuration settings could be stored
in a file (and managed by a configuration management system) and the
configuration file could contain comments.
Proposal:
* IntelMQ gets one global configuration file for all the bots and
the pipeline.conf will be removed
* This global configuration file is
`${PREFIX}/etc/intelmq/intelmq.$ext`. If it does not exists or
does not define any bots, IntelMQ should exit gracefully.
The file extension depends on the chosen format.
* The global configuration file contains an array of bot
configurations with bot-ids as keys.
* Every bot reads the global configuration file and extracts their
own settings (as usual).
* Every bot handles 0 to n `-c /path/to/configurationfile.$ext`
flags, which are treated the same way as the global configuration
file.
The further ahead the configuration file in the commandline, the
stronger the content (this allows us to have multiple non-global
configuration files (i.e. for multiple groups))
Example:
```
> botcommand bot-id -c /etc/bots/botname.$ext -c
/etc/bots/groups/group_foo.$ext
```
* Every bot also consults the environment and the values that are
set their overwrite the values in any configuration file
* There are also configuration files which list settings that are
not bot specific, i.e. via a reserved key default (successor of
the defaults.conf file) or group:id, those are also handled like
other configuration files, but the bot does not compare its name to
the key of the configuration.
All the evaluated configuration formats provide the possibility to
arrange the configuration parameters in hierarchies. To make the
configuration files more readable, IntelMQ should make use of this
hierarchy instead of denoting the different hierarchy levels with
underscores. So instead of writing `http_proxy` the http parameter
would have a childparameter proxy. For backwards compatibility and
cases where the underscore does not imply hierarchy, the underscore
notation will still work. In addition, IntelMQ should also make use
of environment variables - those are still denoted using an
underscore as delimiter and are prepended with `INTELMQ`:
`INTELMQ_HTTP_PROXY`.
[^8]: https://github.com/gethvi/intelmq/blob/ideas/docs/Ideas.md
### Caveats
There are configuration settings, that do not really concern the
bot- for example the type of process manager, that should be used to
run the bot. In an ideal setup, the bot should be totally
indifferent as to if it runs in a Docker container, on bare metal,
in a SystemD unit file or with SupervisorD. This decision should
only concern the tool managing all the bots (intelmqctl or in the
future intelmq-api (which at the moment uses intelmqctl)). Another
example is the enabled setting. At the moment, those are part of the
individual bot configuration, but it might make sense to move them
to a management.conf configuration file which is only for managing
the individual bots, but not for configuring their parameters (this
file would then also (for every bot) have a field that lists the
configuration files the bot should consider when reading its
configuration). On the other hand, this might make the configuration
more complex again, now that we are trying to merge pipeline.conf
and runtime.conf. We could also decide to make those configuration
settings be part of the global configuration file, given that the
individual bots should anyway simply ignore settings they do not
know how to handle.
### Overriding by command line parameters
If needed, a user can override specific bot settings using the -p
switch (i.e. `-p redis_cache=example.com`). This should be easy to
implement, in the best case scenario this is only one line of
additional code in the Bot class.
### Examples
A global configuration file with multiple bots
/etc/intelmq/intelmq.yml
```
- shodan1:
module: intelmq.bots.collectors.shodan.collector
- mylittlebot23:
module: intelmq.bots.expert.asn_lookup.expert
http:
proxy: http://myproxy.tld:80
- fop1:
module: intelmq.bots.outputs.file
output:
filename: /dev/null
```
We can run a bot with intelmq-bot shodan1 which is the same as
`intelmq-bot shodan1 -c /etc/intelmq/intelmq.yml`
Another configuration file with multiple bots
/root/intelmq-bots-managed-by-root:
```
- shodan2:
module: intelmq.bots.collectors.shodan.collector
- fop1:
module: intelmq.bots.outputs.file
output:
filename: /var/log/fop1.log
```
We can run a bot with
`intelmq-bot shodan2 -c /root/intelmq-bots-managed-by-root`;
We can run a bot using
`intelmq-bot fop1 -c /root/intelmq-bots-managed-by-root`
which would then send output to `/var/log/fop1.log`.
A configuration for a group in /etc/intelmq/collector-group.yml
```
- group:collectors
http:
proxy: http://thirdparty.proxy.tld:9000
```
We can run a bot with intelmq-bot
`mylittlebot23 -c /etc/intelmq/collector-group.yml`
which uses the third-party proxy.
## Internal handling
Every bot class defines their own settings as class variables. Every
class variable has to be typed. Every class variable should be set
to a reasonable default, otherwise None. The init of the (abstract)
Bot class should load all the relevant configuration files and then
overwrite the settings. If a setting is still None and the value of
the setting is vital for the functionality of the bot, the bot
should stop and emit a meaningful error message. For the most common
types of settings, there should be Python objects to check the
values. Value checking should only be done after all the
configurations are merged.
--
// Birger Schacht <schacht(a)cert.at>
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
====== IntelMQ Docker Image *(beta)* ======
*CAUTION* Please do not use this image in production yet!
===== Introduction =====
We are constantly trying to make the installation of IntelMQ easier for
a wider user-base.
Therefore we are currently working on an IntelMQ container[^1].
Because of the early stage, input from you is highly appreciated.
We are currently targeting Docker Engine 18.x and higher.
[^1] https://www.docker.com/resources/what-container
===== Current implementation =====
TL;DR: The image is a beta test, docker containers runs isolated, worth
to test!
*ATTENTION* The image provided is not fully Docker compliant.
We are releasing the image to collect feedback and gather information
for future improvements.
The container should be considered as a beta version.
There will likely be breaking changes in future versions of the container.
The image is available on Dockerhub[^4].
You can find documentation on how to install and run the container in
the IntelMQ documentation [^doc]
The container comes preinstalled with IntelMQ Manager[^3], which will be
the only way to control the IntelMQ installation in the container. You
can't use `intelmqctl` outside the container to control IntelMQ in the
container.
At the moment, the image is shipped with the development version of
IntelMQ[^2] and the last stable version 2.2.1 IntelMQ Manager[^3]. This
means that the new IntelMQ API is //not// yet part of the Docker image.
You can check the current included versions by using
```docker inspect --format '{{ index .Config.Labels
"org.label-schema.vcs-ref"}}' certat/intelmq-full:1.0```
Build steps and deployment information is provided in IntelMQ Docker[^5]
Repository.
[^2] https://github.com/certtools/intelmq/
[^3] https://github.com/certtools/intelmq-manager
[^4] https://hub.docker.com/repository/docker/certat/intelmq-full/general
[^5] https://github.com/certat/intelmq-docker
[^doc]: https://intelmq.readthedocs.io/en/latest/user/installation.html
--
// Sebastian Waldbauer <waldbauer(a)cert.at> - T: +43 1 5056416 7202
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Merry Christmas, dear community :)
More or less last minute I decided to do a bugfix release before the
holidays *really* start, because we already collected some fixed in the
last weeks/months. There a no spectacular changes in this minor release,
but the upcoming 2.3.0 will have some major changes for the IntelMQ
Manager backend / the new IntelMQ API and the deprecation of Python 3.5.
Installation documentation:
https://github.com/certtools/intelmq/blob/2.2.3/docs/INSTALL.md
Upgrade documentation:
https://github.com/certtools/intelmq/blob/2.2.3/docs/UPGRADING.md
The deb/rpm packages will be available in the repositories in the next
few hours.
The NEWS:
### Harmonization
A bug in the taxonomy expert did set the Taxonomy for the type
`scanning` to `information gathering`
whereas for the type `sniffing` and `social-engineering`, the taxonomy
was correctly set to `information-gathering`.
This inconsistency for the taxonomy `information-gathering` is now
fixed, but the data eventually needs to fixed in data output (databases)
as well.
There are still some inconsistencies in the naming of the classification
taxonomies and types,
more fixes will come in version 3.0.0. See [issue
#1409](https://github.com/certtools/intelmq/issues/1409).
### Postgres databases
The following statements optionally update existing data.
Please check if you did use these feed names and eventually adapt them
for your setup!
```SQL
UPDATE events
SET "classification.taxonomy" = 'information-gathering'
WHERE "classification.taxonomy" = 'information gathering';
```
The full CHANGELOG:
### Documentation
- Bots/Sieve expert: Add information about parenthesis in if-expressions
(#1681, PR#1687 by Birger Schacht).
### Harmonization
- See NEWS.md for information on a fixed bug in the taxonomy expert.
### Bots
#### Collectors
- `intelmq.bots.rt.collector_rt`: Log the size of the downloaded file in
bytes on debug logging level.
#### Parsers
- `intelmq.bots.parsers.cymru.parser_cap_program`:
- Add support for protocols 47 (GRE) and 59 (IPv6-NoNxt).
- Add support for field `additional_asns` in optional information column.
- `intelmq.bots.parsers.microsoft.parser_ctip`:
- Fix mapping of `DestinationIpInfo.DestinationIpConnectionType` field
(contained a typo).
- Explicitly ignore field `DestinationIpInfo.DestinationIpv4Int` as
the data is already in another field.
- `intelmq.bots.parsers.generic.parser_csv`:
- Ignore line having spaces or tabs only or comment having leading
tabs or spaces (PR#1669 by Brajneesh).
- Data fields containing `-` are now ignored and do not raise an
exception anymore (#1651, PR#74 by Sebastian Waldbauer).
#### Experts
- `intelmq.bots.experts.taxonomy.expert`: Map type `scanner` to
`information-gathering` instead of `information gathering`. See NEWS
file for more information.
### Tests
- Travis: Deactivate tests with optional requirements on Python 3.5, as
the build fails because of abusix/querycontacts version conflicts on
dnspython.
### Known issues
- Bots started with IntelMQ-Manager stop when the webserver is
restarted. (#952).
- Corrupt dump files when interrupted during writing (#870).
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
I finally managed to build the IntelMQ packages for CentOS 8. The
separation into multiple repositories and modules caused some headaches.
You can find the updated installation instructions at
https://intelmq.readthedocs.io/en/latest/user/installation.html
Please report any feedback or bugs here or at
https://github.com/certtools/intelmq/issues
best regards
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear incident handling automation tools list,
dear IntelMQ folks,
First of all, after extensive feedback from many of you, we do have a high level requirements document
for IntelMQ 3.0. It's here: https://github.com/certtools/intelmq/blob/version-3.0-ideas/docs/architectu…
This shall serve as a high level blueprint for IntelMQ 3.0 developments. Sebastian is working on prioritising individual
items for CERT.at and then we will create individual GitHub issues and people (mostly at CERT.at) will be hacking away at it.
Looking forward to this release.
I'll be guiding this release however, I won't be working at CERT.at anymore starting on the 15th of Nov.
Which brings me to an important conclusion:
I thought long about it what we should do when core people leave CERT.at (as in my case, or... maybe Sebastian will leave one day
or get run over by the famous bus which always seems to run over every team member according to manager's expectations ;-) )
In any case, the most solid approach seems to remember what IntelMQ actually is - a **community project**.
It started as one, it is one , it will be one.
In the last years, CERT.at did a lot of the heavy lifting and also a lot of the decisions on IntelMQ's future.
However, with a couple of hundred (600?) installations worldwide, it would be wise to create an **advisory board/architecture board**
for the future developments. I would envision a small-ish group of 4-8 people who take the responsibility of guiding the project for the next ~5 years.
This means:
- staying on top of current developments
- coordinating with the other group members
- coming up with a strategy and procedures (for example, compare with PEP, maybe a lightweight PEP approach is enough)
etc.
- ultimately, guiding the project
It's work, for sure. You should have some passion for the project of course.
I sent out a couple of invite requests to individuals but also would be interested to hear from you, if you would like to participate in such
an effort.
Hence, IntelMQ will become its own entity. And that's good, healthy and ensures a maximum benefit for many users.
If you would like to be on that board, please send me an email.
I'll guide it initially and get everything started.
All the best,
Aaron Kaplan
(private email address for the future: aaron(a)lo-res.org)
PS: we already have one or two companies offering development support for IntelMQ, I would like that they can thrive in this project as well - on a friendly basis. In the long run, this will make the project stronger.
--
// L. Aaron Kaplan <kaplan(a)cert.at> - T: +43 1 5056416 78
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - http://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg
Dear community,
I have two news items for you, both related to documentation:
First, there is an IntelMQ Tutorial, which guides through various
features and tools of IntelMQ.
Lesson one introduces the architecture, concepts and terminology of the
project. Lessons two and three delve hands-on into working with IntelMQ.
Starting with installation and basic usage & configuration they go on to
tackle progressively more advanced topics like using advanced features
or changing the message queue software to be used. Solutions and
explanations are offered for all tasks. In the last lesson you'll learn
how to use intelmq-tools, a third-party software which makes
customization of your IntelMQ instance much easier.
We think that this kind of interactive online documentation is
especially important nowadays when conferences and workshops cannot take
place in real life.
As for all other IntelMQ components, we welcome any contributions and
feedback to the tutorial.
-> https://github.com/certtools/intelmq-tutorial
Second, we have a new IntelMQ Documentation page:
We completely revised the way IntelMQ's documentation is presented:
Instead of single files in the source-code repository, the best place to
read the documentation is now intelmq.readthedocs.io. All pages are
generated using Sphinx, the de facto standard tool for documentation. It
features a better reading experience and a significantly improved
navigation. Furthermore, the new page offers an integrated search as
well as module index covering the complete code documentation
If you find any bugs or have improvements, please let us know!
-> https://intelmq.readthedocs.io/
best regards
Sebastian
--
// Sebastian Wagner <wagner(a)cert.at> - T: +43 1 5056416 7201
// CERT Austria - https://www.cert.at/
// Eine Initiative der nic.at GmbH - https://www.nic.at/
// Firmenbuchnummer 172568b, LG Salzburg