thecrowler

Repository on GitHub, which provides public git repositories with URLs.

Repository on GitHub, which uses git. git can track the changes, who made them, and when they were made.

The project includes interim releases (and also pre-releases for testing) as well as having an official Develop branch that is used to push code that needs also human testing, not just automated. On top of all this the repository has plenty of automated quality tests, the code comes with unit tests which are mandatory also for new features and has also automated coding style checks.

Repository on GitHub, which uses git. git is distributed.

The project does have unique versions numbering and we version also RCs when and if available.

Developers and contributors push their PRs against Develop branch, not Main (Main is protected and requires a PR from Develop with approval). Devs and contributors PRs against Develop are tested automatically and reviewed by humans before code gets merged into Develop (same happens for code that gets merged from Develop to Main). Devs and Contributors are also required to install pre-commit which is configured to run a lot of tests (and run also all unit tests) at every got commit. When code has passed ALL automated tests and human reviews and merged in Main, then an RC tag is emitted for testing on 3rd party systems. an RC has a period of a month, to give time to user to test. If no issues are reported, then the Main gets released again with the final release number

Checks in pre-commit here: https://github.com/pzaino/thecrowler/blob/main/.pre-commit-config.yaml Tags here: https://github.com/pzaino/thecrowler/tags

Project is new, so not official releases has been emitted yet.

Each release (and pre-release) have a fully detailed release note: https://github.com/pzaino/thecrowler/releases/tag/v0.9.3

Full changelog example: https://github.com/pzaino/thecrowler/compare/v0.9.2...v0.9.3

The release note will indeed include all publicly known run-time vulnerabilities when we'll start releasing production versions.

https://github.com/pzaino/thecrowler/issues

We use GitHub Issues number to track each individual issue

We aim to have ALL found issues public (we use GitHub for users to report them) and we acknowledge them all as soon as we are able to reproduce, so it could be even minutes.

We do our best to respond to enhancement requests as fast as possible, but the project if fully free, so involved people also need to ensure they do their jobs first to then have time for the project.

We use GitHub issues and Discussions, so everything is public. https://github.com/pzaino/thecrowler/issues?q=is%3Aissue+is%3Aclosed

https://github.com/pzaino/thecrowler?tab=security-ov-file#security-policy

Private vulnerability reports are supported and they can be submitted via email to the project author.

We check the project GitHub and the emails every day. So, every vulnerability found and reported will be checked certainly within 14 working days.

The project provides a working build system to rebuild the entire set of micro-services at once:

https://github.com/pzaino/thecrowler/blob/main/docker-rebuild.sh

The project uses Docker and Docker compose to create containerized builds of all the required components and it also offers build scripts that add platform detection so to help Docker to pull or build containers appropriately.

https://github.com/pzaino/thecrowler/blob/main/docker-compose.yml

https://github.com/pzaino/thecrowler/blob/main/docker-build.sh

The project IS buildable using ONLY FLOSS tools, to build it one needs go lang only. Database is based on PostgreSQL and SQLite, components are packaged in Docker images at build time and using freely available tools. It works fine with Docker provided with Linux Distribution (aka doesn't requires to install docker from docker.io)

https://github.com/pzaino/thecrowler/blob/main/docker-build.sh

https://github.com/pzaino/thecrowler/tree/main#build-from-source

For automated tests: https://github.com/pzaino/thecrowler/actions

A user can run go test to run unit tests locally on their system and at any time.

The entire suite of tests covers everything. And we are also adding system tests to measure performance under stress.

The project allows to do continuous integration. Contains can be built off-line and replace existing containers too.

From the CONTRIBUTING.md file:

"If you've added code that should be tested, add tests."

link here: https://github.com/pzaino/thecrowler/blob/main/CONTRIBUTING.md

https://github.com/pzaino/thecrowler/blob/main/doc/test_policy.md

It is:

"If you've added code that should be tested, add tests. For more information on testing, see Test Policy for TheCROWler."

From the CONTRIBUTING.MD file, link here: https://github.com/pzaino/thecrowler/blob/main/CONTRIBUTING.md

Project is written in go (and gofmt) and pre-commit forbid a developer from even being able to push their code in a develop branch. tests are executed even when code compiles fine and we have all go lang warning.

As mentioned, we do address warnings.

We use Go lang and gofmt as well as: - go-fmt - no-go-testing - golangci-lint - go-unit-tests

https://github.com/pzaino/thecrowler/blob/main/.pre-commit-config.yaml

We do have a primary developer with multiple courses and more than 30 years of experience in coding and secure coding.

Our primary developer works in the Cyber Security field and has experience in developing and designing IDS, IPS, Firewalls and Anitvirus software and deals daily with vulnerabilities and CVEs, CWEs and CPEs

Software uses standard HTTPS for the few cryptographic elements and for that uses the standard go lang libraries.

We also use a vulnerability bot that automatically creates pull-requests with updated dependencies when one is found having a vulnerability.

we check for private credentials from the developer local system using pre-commit and re-check on GitHub as well

SonarQube (locally), CodeQL, Codacy on GitHub.com

We also use SonarCube LINT in VSCode studio so security issues are detected as we write code (real time), we check again with sonar-scanner before commit and with CodeQL and Codacy at every commit on gitHub

project is fully written in Go Lang, so no memory-unsafe code.

Not only we do, we also generate our containers hardened and code is executed with low privilege accounts inside the containers. Where applicable containers have also read only filesystem enabled.